Welcome to TiddlyWiki created by Jeremy Ruston; Copyright © 2004-2007 Jeremy Ruston, Copyright © 2007-2011 UnaMesa Association
Roster: Sheelagh Carpendale (SC), David Ebert (DE), Brian Fisher (BF), Jean Scholtz (JS), Eric Alexander (EA), Bongshin Lee (BL), Min Chen (MC), David Kirsh (KD), Wenwen Dou (WD), Kirsten Winters (KW), Remco Chang (RC), Alex Endert (AE), Zhibin Lei (ZL), Melanie Tory (MT), Matt Brehmer (MB), Lauren Bradel (LB)
!Introductory session
*SC: 7 scnearios; algorithms eval, visual quality assessment
*JS: trust in algorithms as eval
*DE: quality of assessment re: signal processing
*BF: cognitve pscyhologists will say: "you're not qualified to use qualitative social science methods"; air traffic control field work; Multiple Token model, Fechnerian model, used prescriptively; Egon Brunswick on individual differences
*DK: efficiency vs. effectiveness: e.g. the efficient shoe factory who produced outmoded shoes; learnability and expertise (time to learn); transformability: new methods
*DK/AE: picture of the apple vs. a magnifying glass to move around the apple; passive vs. active probing; medical student study of visualization use vs. debrief lecture: time to decision was longer, but comprehension greater; as-hoc vs. non ad-hoc concepts, highly-situated vs. non highly-situated concepts; some are more task-dependent
*MC on prior evaluation work (BELIV, Science of Interaction workshop); VIS 2012 panel on visualization quality assessment; -isms in psychology, visology? selective attention, visual search
*BF: to what extent is visual ability plastic? cyber psychology; e.g. Asian city dwellers vs. westerners vs. Asian rural dwellers vs. westernized Asians (RC: vs. Las Vegas dwellers)
*SC: different design schools: design (Tufte, Few), semiotics (Bertin), applied cogsci / stats (Cleveland / Wilkinson); so how to evaluate vis from different schools of design? visualization a pre-Newtonian science, elements not yet known; etymology of "spell" eminating from druidic practices
!Working group on //Empirical work before and during the design process for visualization//
SC, MT, BL, MB
!!Challenges
!!!Before
*access to data
*finding the right collaborators & domain experts
*people canഥll you what they need
*finding the right immersion strategy (fly-on-wall, shadowing, moment in time, apprenticeship, etc.)
*setting ownership / expectations for paired interdisciplinary research
!!!During
*Users not criticizing the design - once they see something too polished they wonࣲiticize, assuming a prototype is a finished product
*need some sort of prototype
*heuristic evaluations can be used but they make you be conservative
!!Methods
*apprentice
**working with an లentice�uld be an accepted and promoted approach
**not well understood, not done enough, hard to publish
**does a లentice�d to have domain background?
*participant observer
*shadowing
*artifact analysis
*fly on the wall
!!!Analysis methods
*(open coding, semi-structured interviews, etc.)
!!Community comments / Problems with the community
*paper is rejected because there is no separate evaluation - because expert is part of the design team
*ﵠdidnॸplore enough of the design space"
*community does not understand empirical research, doesn૮ow methods well enough
**and community canలoperly review it - distinguish good work from poorly done work
!!Community Challenges
*To bring increased awareness and acceptance of empirical research. Particularly the enormous variety of what rigor is in the different methods.
*To bring change to the review and publication process. E.g. remove bad studies from study / technique papers.
*Change the call for papers to encourage empirical work which details processes at early stages of visualization design.
!!Impact / Value
*worst case gets you requirements analysis
*find out details about the f the process. It is usually the how of a process that makes it useful/acceptable/adopted
*if one wants to be grandiose about claims it is these types of immersive design processes that can lead to transformative results
!!Ideas:
*paired research
*more apprentice work
*community doesne all the relevant methods examples because they堰ublished outside of VIS. Website of papers?
*summary article of types of లentice㴵dies - different variety of approaches that are possible 죡ll-to-action�er
!Debrief session - day 1
*DK/EA/JS/KW on field studies
*RC/BF/LB on lab studies
*WD/MC/AE/ZL on evaluating mixed initiative systems
*DK: eval-by-task-type taxonomy?
*BF: scene perception literature only deals with naturalistic scenes, not artificial scenes
*RC: lab studies aimed at integration with cognitive models: ACT-R, SOAR, Math+Pscyh
*RC/SC: Toy datasets / toy examples as generalizable sources of data; VAST challenge dataset a toy dataset?
*BF: Herb Clark re: mixed initiative system efficiency
*MC: continuous evaluation of mixed-initiative VA systems: e.g how many times did human correct system? how long spent correcting?
!Day 2: Critical application areas
Empirical work at initial stages of the visualization / VA design process
!!Why?
These days people are often overwhelmed with their data; they do not know how to start, do not know what is possible, and cannot tell you what they need. Since their situations vary and complex, traditional empirical methods and design practices no longer suffice. Visual Analytics (VA) designers need to integrate their empirical work deeply into their initial design processes.
!!Challenges
By its very nature any situation that will benefit from visual analytics is complex. There is a growing awareness that to approach understanding of complex situations, such as ecosystems, at least some of the studying must look at the system as a whole. Thus if we want to design and develop a visual analytics system for a given situation, many of the common types of empirical methods in which one selects and closely examines some aspects of the situation are not sufficient. Thus we need to find, adapt, and develop new methods for these needs. Specific challenges include:
*Design goals can be quite varied, particularly when we diverge from traditional Vis & VA. It is not clear which of these may be most important as design goals in any given situation, so we need ways to discover the design goals. Once the design goals are determined, there may not be established instruments to measure baseline and change effects (e.g., how can one measure attitudes towards energy conservation).
*Prospective users having domain expertise are often remote and it is therefore not straightforward to apply observations. New methods are needed to obtain rich data from remote participants.
*Diversity of situations: methods such as interviews and "fly-on-the-wall" observation may not be appropriate in all contexts, such as hospitals, where it can be too intrusive to be there. We need ways to obtain rich data (like one would obtain from observations) but with less intrusive means.
*Differences in individuals including cultural differences, age, gender, physical capabilities, etc. are often ignored. But, they may require different design goals, means to collect data, and ways to interact with them.
!!Application Area: Personal Visual Analytics
Prior to the design of visualization or visual analytics systems, practitioners require opportunities to immerse themselves in environments where they may observe current data exploration, analysis, and decision-making processes. However, personal data management and analysis is often sensitive and private, so it can be too intrusive to "be there", to continuously observe individuals as they perform these processes, or to make video recordings. For instance, consider the sensitivity of personal finance or health information, or the intrusiveness of observing someone in their home environment. Given this constraint, there is a strong reliance on methods such as self-reports, software logging, artifact collection, and diaries; some data may need to be anonymized. These methods may not provide data that is as rich as direct observations, though when used in conjunction, a partial reconstruction of a process may be possible.
Personal visual analytics systems are often targeted at special populations. Consider applications for diabetics that monitor one੮sulin levels, or applications for elite athletes that monitor caloric intake and expenditure, or those that monitor training regimes. Access to individuals from these special populations can be limited, and is similar to the limited access to individuals with domain expertise in other application areas; they may be remote or it may be difficult to observe individuals in the contexts where these processes are performed.
Personal visual analytics also has a wide variety of design goals, and traditional goals such as efficiency may rarely be the most important. For example, if a visualization of home energy use is realized as an ambient display in oneving room, fitting into the aesthetic of the room may be a critical goal. Design goals may also vary considerably among individuals. While some people may track their physical activity to meet a personal objective (e.g., weight loss), others may do so simply out of curiosity or to share with their friends. We need ways to discover the diversity of design goals so that we can design systems to accommodate this diversity.
!!Impact / Value
The advantages of early design empirical research include: improving understanding of the situation; learning about nuances in behaviors and processes; providing opportunities to design appropriately for the situation; and, quite significantly, providing a baseline for future comparisons.
It is still unusual to put this type of emphasis into an early design empirical approach. These types of approaches are in their infancy; however, indications are that these types of approaches may be key. For example, leading thinkers (including B. Buxton) have often pointed out to the community that too much attention is currently given to getting the design right (to evaluating and improving a given design) and not enough attention is given to getting the right design (making sure that the design is appropriate for the given situation). These types of empirical design processes will lead to us as a community being better equipped to getting the right design.
!!Abstract
A post-deployment evaluation of a visualization tool can be difficult to conduct, particularly when evaluation criteria is contingent on determining how domain-specific professionals use the tool in the context of their ongoing work. Remote collaborators and users add to the logistical complexity of these evaluation studies. Such is the case with Overview, a visualization tool for exploring large document corpuses, built by our collaborator at the Associated Press. In this report I reflect upon the process and findings of an ongoing post-deployment, mixed-method evaluation of Overview, which includes an in-depth case study of a journalist who used Overview to investigate a large email corpus. I also reflect upon how this work factors into my long-term research goals relating to exploratory data analysis and evaluation in information visualization.
!!Introduction and Background
An evaluative study or experiment helps us to gauge the efficacy of a visualization tool or technique. The prominence of evaluation in information visualization research has grown in recent years.
A recent survey (<<cite Lam2011 bibliography:Bibliography>>) of over 800 articles published between 1995 and 2010 at four major visualization venues reported an increasing trend of papers containing an evaluation component. The vast majority of these evaluation studies focus on usability or graphical perception issues. However, there has yet to be a similar increase in the number of papers reporting an evaluation of visualization users' context-dependent processes.Evaluating users' processes with and without visualization can be highly informative at both formative and summative stages of visualization tool development (<<cite Andrews2008>>, <<cite Ellis2006>>). Yet these evaluation studies tend to be time-consuming and costly, posing many logistical challenges, particularly when collaborators and users outside of academia are involved (<<cite Sedlmair2011>>). As a result, post-deployment or summative evaluation studies are rarely published, and often report negative results (<<cite Gonzalez2003a>>, <<cite Munzner2011a>>).
Novel evaluation methodologies and methods are emerging to overcome the challenges associated with studying user processes. In 2006, a workshop (<<cite Bertini2008>>) was established to discuss visualization evaluation //beyond time and error//, the historical metrics of usability and graphical perception studies. These metrics cannot be used to reliably answer our process-centric evaluative questions, nor can we rely upon controlled laboratory settings, prescribed experimental tasks, or research participants who are unrepresentative of an intended user population (<<cite Carpendale2008>>, <<cite Plaisant2004>>, <<cite Sutcliffe2000>>). Instead, the novel evaluation research community has established alternative metrics, conducting studies in externally valid research settings with representative users. This community has proposed methods for measuring gains in insight (<<cite Saraiya2005b>>, <<cite Saraiya2006>>), for comparing problem-solving strategies (<<cite Mayr2010>>), and for assessing how visualization tools can facilitate multiple forms of learning (<<cite Chang2010>>). They have demonstrated that evaluation methodologies that incorporate qualitative data collection and analysis methods adopted from the social sciences can be rigorous, inductive, and replicable (<<cite Isenberg2008>>, <<cite Kang2011>>, <<cite Tory2008>>). Finally, they have shown that valid and reliable evaluation research often requires time, patience, and longitudinal coordination with collaborators and users (<<cite Lloyd2011>>, <<cite Saraiya2006>>, <<cite Shneiderman2006>>).
The work I discuss in this report, a preliminary evaluation of a visual document mining tool, has been motivated by my interest in the proliferation of emerging evaluation methodologies and methods. I am also motivated by my interest in how visualization users perform exploratory data analysis: I want to know how visualization is used to solve ill-defined problems, to discover and understand, rather than to lookup and verify (<<cite Marchionini2006>>). One way I like to describe this process is how a visualization user may not know a priori what they are looking for in a dataset, but they'll know it when they see it. I want to be able to reliably evaluate how a visualization tool supports this process of serendipitous discovery.
A fair question at this point is one of granularity: how do you define the process of exploration and discovery? There exists a body of work that characterizes these processes from an abstract, top-down perspective of human cognition (<<cite Amar2004a>>, <<cite Pirolli2009>>, <<cite Springmeyer1992>>, <<cite Valiati2006>>), while others have observed and characterized these processes from a bottom-up perspective, at the level of interface events (<<cite Amar2005>>, <<cite Shneiderman1996>>, <<cite Valiati2006>>, <<cite Winckler2004>>). Task taxonomies constructed either on abstract models of human cognition or on interface-level events can be helpful, but all too often they can be stifling if they are meant to be used as evaluation criteria (<<cite Beaudouin-Lafon2004>>). This is particularly true when one is required to compare how users interact with different visualization tools across domains.
A valid and comparative evaluation methodology requires a robust mid-level task characterization of exploratory data analysis, one that spans domains and tool interfaces.
My long-term goal is to contribute to the construction of such a task characterization. To do this, I will study exploratory data analysis as it occurs across multiple domains, the tools individuals use, and the processes they undertake, while assessing how these tools and processes succeed or fail. I will employ a repertoire of established and emerging data collection and analysis methods, while reflecting upon the efficacy of my methodological choices. This endeavour began in March of this year, studying journalists' use of a visualization tool for exploring large document corpuses.
!!!The Overview Project
In recent years, many large corpuses of emails, reports, and other documents have been "dumped", "leaked", released, or declassified by corporations, government agencies, and other organizations. A well-known example is that of WikiLeaks, an organization that released nearly 400,000 documents relating to the conduct of armed forces and security contractors during the recent war in Iraq. Since that time, journalists have [[reported|http://goo.gl/qqGGh]] on what was contained in this corpus, which included startling patterns of civilian casualties, friendly fire incidents, and observed breaches of protocol. My goal has been to better understand how journalists explore or "mine" these corpuses, how they seek information to support or refute prior evidence, or how they come to discover unanticipated newsworthy stories hiding in the data.
[[Journalism is a field in transition|http://goo.gl/64Yh2]]. [[Areas of specialization and experience among journalists are changing|Data Journalism: Links and Literature]], reflecting the shift toward online content presentation and the necessity to address the growing amount of structured and unstructured information at one's disposal. As a result, it is difficult to predict how and when a data visualization tool will be used, who will be using it, and whether it will be an effective part of the process of writing a convincing news story. These were the motivating questions of JS, our collaborator at the Associated Press, who recently worked with our group to develop Overview, a visualization tool to support the process of mining large document corpuses (<<cite Ingram2012>>). A prototype of [[Overview|http://www.overview.ap.org]] was released publicly in early 2012. Overview's user interface, shown in the accompanying figure, is comprised of set of linked views for exploring hierarchical clusters of related documents within a corpus, providing means for reading documents, as well as for tagging them with personally meaningful terms or phrases.
[>img(50%, )[Figure 1. A figure produced using Overview, displaying Wade�l corpus: (a) a topic tree displays hierarchical clusters of related documents, (b) document tagging functions and a list of user-generated tags, (c) a scatterplot of documents in the corpus; nearby documents are similar, (d) a list of documents having the currently selected tag in (b), (e) an embedded document viewer displaying the currently selected document in (d).|http://cs.ubc.ca/~brehmer/research/rpe/overview_jw.png]]
JS believes that the need to mine large document corpuses will increase in the coming months and years, and that current practices made this process impractical. These practices gravitated to either keyword searches or brute-force approaches: reading or skimming all documents in a corpus. The former approach requires one to know a priori what one is looking for, while the latter is too time-consuming, difficult to streamline and manage.
In both cases, exploratory analysis is poorly afforded: it is impossible to sample representative documents in a corpus and extract the trends or patterns. Overview has been designed to make this exploration possible.
A post-deployment evaluation of Overview was an attractive opportunity for both JS and our research group. The project was also well-situated within my larger research goals relating to evaluation methodologies and the characterization of exploratory data analysis.
In the sections that follow, I describe the methodology of my evaluation, followed by my current findings. Working with a collaborator from outside of academia has been a novel experience; in the final section of this report, I reflect upon the advantages, constraints, and implications of the //collaborator-as-gatekeeper// relationship. I then look back upon my research process thus far, discussing perceived inefficiencies as well as ideas for future improvement. I conclude with thoughts on how this project fits into the larger scheme of my ~PhD research. I do not yet claim a defined research contribution, as this paper is intended to be a largely reflective account of an ongoing project.
!!Evaluating Overview
My intent of conducting a post-deployment evaluation of Overview has been to assess whether or not it meets the exploratory data analysis needs of target users, individuals with large document corpuses and hunches about potential stories contained within them; could Overview make writing a story possible in situations where doing so was previously impossible, or at least highly impractical? An evaluation would also serve JS's need to identify major usability issues and barriers to user adoption. Finally, given that Overview displays a document corpus using novel techniques (<<cite Ingram2012>>), our evaluation would also seek to determine if users had a conceptual understanding of how these visualizations represented the structure of semantic relationships in a document corpus.
!!!Methodology
An ideal research methodology for this work would include in-situ interview and observation sessions, in the spirit of longitudinal insight-based evaluations (<<cite Saraiya2006>>) and multi-dimensional long-term case studies (<<cite Shneiderman2006>>). However, our user base is distributed around the world, and logistical constraints keep me from visiting each individual journalist in their newsrooms. Furthermore, like many busy professionals, journalists are often working under tight deadlines, and have little time to participate in multi-session longitudinal studies. As a result, observing and interviewing these journalists in-situ is infeasible.
These constraints limit possible data collection methods, which in turn has dictated my choice of methodology, a less than ideal situation. Relying upon retrospective accounts of journalists' experiences with Overview, via in-depth interviews and elicited textual accounts, necessitates an interpretive perspective, a focus on the language journalists use to describe their processes.
A grounded theory methodology (<<cite Charmaz2006>>) appeared to be appropriate, given a need to construct a post-hoc interpretation of journalists' processes of exploratory data analysis. The constant comparative philosophy of grounded theory prompts me to think flexibly, to make comparisons between the process of using Overview with other journalistic processes, with processes relating to exploratory data analysis as it occurs in other domains. Comparisons are also made between journalists and over periods of time, before and after the introduction of Overview into one's workflow. I initially proposed this methodology in an [[earlier research proposal|Document mining in data journalism]], which was submitted as a final project for a [[course in interpretive and critical research|EPSE 595]], taken in the winter term of 2011/2012.
I am not the first to adopt a grounded theory methodology and its methods in visualization research. The methodology has informed prior work characterizing the use of visualization by professionals in domains such as architecture (<<cite Tory2008>>) and intelligence analysis (<<cite Kang2011>>), as well as processes of high-dimensional data analysis dimensionality reduction across multiple domains (<<cite Sedlmair2012a>>). In evaluation research, a "grounded evaluation" methodology has been used for the in-situ study of a visualization tool's efficacy in a target context of use (<<cite Isenberg2008>>).
A further justification for the use of a grounded theory methodology is that my research questions are not theoretically deduced hypotheses, nor is my objective to explicitly validate or refute prior task characterizations of data analysis (<<cite Amar2005>>, <<cite Pirolli2009>>, <<cite Springmeyer1992>>). Rather, my eventual goal is to construct a mid-level, cross-domain task characterization. Thus my starting point is not a rigid theoretical framework, but with the personal accounts of Overview users. Of course, no research exists in a void, uninfluenced by previous work. As such, my research questions are informed by assumptions and sensitizing concepts held within the visualization community. It is these sensitizing concepts that have allowed me to frame data collection methods, particularly a preliminary set of interview questions.
These sensitizing concepts include the notion that exploratory data analysis occurs in stages (<<cite Pirolli2009>>). Exploratory data analysis may involve stages of hypothesis generation, without a well-defined set of questions in mind. There are also stages of hypothesis validation, where the goal is to support or refute prior evidence. Individuals may or may not engage in both types of stages during the course of a single investigation.
The products of exploratory data analysis are also among my sensitizing concepts. These products include moments of insight (<<cite Chang2009>>, <<cite North2006>>, <<cite Yi2008>>), serendipitous discoveries (<<cite Andre2009>>), and both optimal and suboptimal solutions to closed- and open-ended problems (<<cite Mayr2010>>). Admittedly, these products are ill-defined constructs within the language of our research community, and it is necessary for me to attain our participants䥲pretations of their meanings, as well as their own terminology.
Finally, these sensitizing concepts include the disentanglement of a data analyst's learned expertise (<<cite Chang2010>>). By this I mean that an analyst may have expertise related to their domain, expertise using specific analytical tools or techniques, and expertise regarding the data under examination, its semantics, and its provenance.
!!!!Recruitment of Participants
The recruitment of research participants has been difficult to predict, and dependent on the number of individuals who download, install, and use Overview for the purpose of writing a story. As a representative of a reputable news agency and a recipient of a prestigious Knight Foundation grant, JS's clout meant that Overview would be highly visible to the journalism community. Participant recruitment is therefore a matter of waiting for prospective Overview users to establish contact with JS. This means that he acts as a gatekeeper to research participants, an aspect of this project that I reflect upon later in the discussion section.
!!!!Data Collection Methods
My primary data collection method is that of an in-depth, open-ended interview, recorded for later transcription. Following the methodology of grounded theory (<<cite Charmaz2006>>), I have not specified the number of interviews that I planned to conduct a priori. The final number of interviews will depend upon how many Overview users express a willingness to participate in interviews and upon whether and when data saturation occurs, which I discuss in the following section.
Multiple interviews with each participant would be ideal, as processes change over the course of an investigation. However, this is an unrealistic plan; as mentioned above, journalists often conduct their investigation and write articles under tight deadlines, and often only have the spare time required to commit to an intensive interview after their story is written.
Regarding the content of these interviews, I began with a short list of interview foci and a small set of representative questions for each, composed according to guidelines for open-ended interviews (<<cite Fontana1994>>) and for interviews conducted in the context of a grounded theory study (<<cite Charmaz2006>>). These foci correspond with the sensitizing concepts described above. I ground the interview in the current document corpus under investigation, inviting comparisons from the interviewee's prior body of work, before using Overview. [[This list of foci and questions|Overview Deployment Field Survey]], is flexible and subject to change, owing to the possibility that early interviews illuminate unanticipated concepts.
[>img(25%, )[Figure 2. An excerpt of Overview易file, listing timestamped events.|http://cs.ubc.ca/~brehmer/research/rpe/overview_log.png]]
I compliment interviews by gathering texts and other information from participants, such as Overview's log file of timestamped interface interactions. I also gather information regarding their data, such as the format and number of documents contained in the corpus under investigation. Finally, I request copies of notes participants take during the course of their investigation. Journalists' notes are less intrusive than dedicated progress or insight reports (<<cite Rester2007a>>, <<cite Saraiya2006>>); they are personalized and externally valid, offering yet a another window into their own interpretations of their investigative process.
!!!!Data Analysis Methods
Data collection and analysis occur concurrently. I subject interview transcripts and textual artifacts collected from journalists to multiple iterations of coding, wherein I call upon the constant comparative method, the basis of grounded theory. First, //initial coding// labels excerpts from transcripts and notes with words from the participant's own vocabulary, using an active voice to emphasize a focus on a process occurring in time (<<cite Charmaz2006>>). Next, I generate tentative categories of codes, each with an explanatory rationale based on comparisons between code instances, recorded as memos. This process informs subsequent data collection and the process of //focused coding//, wherein I re-code the data using emerging categories.
As categories are refined and theoretical concepts emerge through the process of focused coding, I may reach a point of theoretical saturation, when no major unexamined concepts are expected to appear as a result of future data collection. At this time, a mid-level task characterization of exploratory data analysis in this domain may begin to emerge. This stage will also invite comparisons between my nascent theory and existing theories (<<cite Amar2005>>, <<cite Pirolli2009>>, <<cite Springmeyer1992>>).
Extensive log file analysis is complimentary to the analysis of transcripts and textual artifacts, providing me with a partial yet objective account of usage strategy (<<cite Pohl2010>>). After parsing, aggregating, and filtering events in the log, I can extract descriptive usage statistics. The log file, as shown in the accompanying figure, reveals how many documents were viewed and tagged during the course of a user's investigation, as well as how various user interface features were used over time.
!!!Findings
To date, two professional journalists and a pair of academic researchers have completed an analysis of a large document corpus using Overview. I am also aware of several additional journalists and academic researchers who may be currently using it. Finally, I am aware of a journalist, an academic researcher, and business consultant who abandoned use of Overview, as it either did not meet their needs or was incompatible with their existing workflow or set of tools.
The discovery of prospective users from fields outside of journalism was unanticipated, indicating that Overview may support exploratory data analysis in the digital humanities, communications, and related domains.
!!!!Case Study: The Tulsa Journalist
Of the two journalists who completed an analysis of a document corpus using Overview, one published his [[findings|http://tulsaworld.com/tpdtechemails]]. JW, a //Tulsa World// staff journalist was willing to be interviewed, and he provided us with not only his log file, but also his entire dataset. He also kept thorough notes during his investigation, and would later write a [[blog post|http://goo.gl/QJ64R]] intended for prospective Overview users. In many ways, he was an ideal research participant, and I should not expect the same amount and depth of information from all future participants.
Beginning with an anonymous tip and hunch relating to a botched, $4 million police equipment purchase, JW accumulated a document corpus of 6,000 Tulsa Police Department emails via a municipal records request. Using Overview, the journalist discovered newsworthy evidence contained in only a handful of emails: several police officials were responsible for the poorly managed purchase, and were caught in a potential conflict of interest with an equipment supplier.
I interviewed JW 3 days after his story ran. The 100-minute interview was conducted via a Google+ Hangout video chat, a service that also affords chat participants the ability to share their screen. This feature permitted JW to walk us through his process, both with and without Overview. Using a screen capture application, I recorded JW's video feed along with the audio conversation. I later transcribed this interview, whereupon I realized that the ratio of time spent transcribing to interview duration was approximately 4:1. I then coded this transcript, alongside JW's notes, according to the initial coding scheme described above. Meanwhile, my analysis of JW's log file provided an objective account of his process, corroborating with his subjective, retrospective description. With over twenty thousand events logged, log analysis was exhaustive yet time-consuming, requiring the better part of a week to complete; I reflect upon the utility and duration of log analysis in the following section.
The 27-year-old JW has only been on the Tulsa "cops beat" for a couple of years. While he considers himself to be tech-savvy, he has no formal background in programming or visualization use. As such, he required a considerable amount of assistance while installing and configuring Overview. This story was, in his words, the biggest story of his early career. The only similar story in his prior body of work was one about the emergency response practices of a local college security force, an investigation that necessitated the examining of a 2-foot high stack of emergency call log printouts, while making annotations with highlighters.
[>img(30%, )[Figure 3. The median time spent viewing a single document was lower during sessions in which more documents were viewed.|http://cs.ubc.ca/~brehmer/research/rpe/overview_jw_results.png]]
What was most fascinating about JW's process was his determination to read and tag most of the documents in the corpus. In 56 hours of use, spread over 15 non-consecutive days, an impressive 70% of the documents in the corpus were selected and viewed in Overview's embedded document viewer for at least one second. Rather than use Overview as a means of broadly exploring and sampling documents within a corpus, JW's usage strategy defied expectation:
>//"At the worst I wanted to use [Overview] as a way of organizing me looking through every email, and at best I wanted to look at most of the emails."//
A systematic and efficient process was a recurring theme in the interview: the quick dismissal of uninteresting or irrelevant documents, the worry about overlooking important documents, the prevention of having to re-read documents, and the vigilant scanning for unique or "rogue" documents. Always conscious of deadlines quotas, JW wanted to streamline his process of viewing documents as much as possible, at times referring to his rate of skimming individual documents as a "speed test":
>//"The speed factor, you're talking about just clicking and glancing, it could literally be as fast as 3s per email. Until I got to one that I needed to knowﴻ//
The accompanying figure shows that during the first several sessions of Wadestigation. Longer median document viewing times reflects the process of getting an initial feel for the documents contained in the corpus. During the longer sessions that followed, shorter median document viewing times reflected his streamlined, exhaustive scouring of the corpus. While finalizing his investigation, fewer documents were read for longer periods of time; returning to a smaller number of significant documents.
By this time he was preparing to write his article.
JW's similarly completist use of Overview's document tagging feature may have implications for information management in exploratory data analysis tools. He created twenty-two tags, tagging 92% of the documents in the corpus with at least one of them. He explained that he became "obsessed" with having every document tagged. This came at the expense of having a complicated, cross-cutting, and disorganized set of tags. Regarding his tags, it was clear that eighteen of them reflected the content of the documents: email subject, sender, or recipient. The remaining four tags reflected a document's level of importance (//Important, Trash//) or personal memo for follow-up (//Check on// [this], //Weird//). He explained that he would have additionally liked to have had //unread / read// and //relevant / not relevant// flags for each document. The latter four tags and flags cross-cut the eighteen content-based tags, and suggest that two strategies of information management were being used during his investigation.
All things considered, could JW have carried out his investigation without Overview? He admitted the possibility, however it would have taken an estimated four months of full-time dedication to read the entire corpus, while maintaining the same level of organization that Overview provided him. By contrast, his investigation using Overview took less than two weeks. With news agency deadlines and article quotas to consider, a longer-term project would have been relegated to a part-time assignment; upon its completion, the story would have run the risk of no longer being newsworthy.
!!Discussion
The post-deployment study of Overview is far from over. At present, I have only spoken in depth with a single user, one who may not be representative of all Overview users. Finding users with different goals and strategies will take time, patience, and a reliance upon JS as a collaborator. In the meantime, I can reflect upon what I've learned and refine my processes of data collection and analysis.
!!!Samplers and Completists
Overview was built for the purpose of broad exploration, sampling documents in a corpus as a means to extract trends or irregularities.It came as a surprise to both JS and myself that Overview would instead be used for performing an exhaustive and systematic search.JW's search criteria was approximate, as opposed to exact, necessitating exploration rather than processes such as keyword search, browsing, or navigating.
Broad sampling and exhaustive approximate search are two usage strategies, both being variants of exploratory data analysis.
I presented a comprehensive analysis of JW's use of Overview to JS, whereupon we conjectured that JW's usage strategy reflected his initial records request, which was filtered a priori to emails with specific senders, recipients, and subject lines. His task was to find the small number of "smoking gun" emails, those containing the evidence needed for his story, validating the hypotheses that emanated from the original anonymous tip.
A sample-based usage strategy might arise in cases where the document corpus is "leaked" or "dumped", rather than requested. In the case of document leaks such as the [[Iraqi war logs|http://goo.gl/qqGGh]], a journalist may explore the corpus broadly, reading far less than 70% of the corpus in an attempt to attain a gist or //overview// of its contents, rather than seek the "smoking gun". Instead, JW was a completist, one who used Overview as a tool for performing an exhaustive, "I have to read everything" investigation, albeit more systematically and efficiently than that of his earlier story involving the 2-foot stack of call log printouts. It also remains to be seen if strategies of information management via tagging used during an exhaustive investigation are also called upon by those using Overview for a more sampling-based analysis of a document corpus.
At this point, I would like to interview an individual who uses Overview with a sample-based strategy, preferably one working with an unfiltered document corpus emanating from a dump or leak. Should they have an intention to determine the major trends or themes within, I would be curious to compare their process against that of JW's.
!!!The Collaborator as Gatekeeper
Collaborating on a project with non-academic visualization tool builders has well-known advantages and disadvantages (<<cite Sedlmair2011>>). In my case, a robust, publicly-available tool was completed and deployed before the evaluation project began. As mentioned above, JS's professional visibility would also attract potential users within the journalism community. As a junior visualization researcher, I do not have sufficient clout within the journalism, digital humanities, or related communities to attract users. It is furthermore inappropriate to evaluate the utility of Overview with undergraduate student volunteers or those not working in domains that do not regularly encounter large document corpuses. As a result, our collaborator is the gatekeeper to prospective research participants.
But is reputation enough to attract users? In the six months since Overview's launch, we are aware of only a handful of individuals that have used it, with only one journalist using it to write a story. The time commitment of mining large document corpuses is extensive, not including the time required to install, configure, and learn how to use a new tool such as Overview. These problems may be alleviated in a forthcoming release of the Overview web application, which will feature a simplified user interface and a reduced feature set. It will also eliminate the need for a desktop installation, requiring less initial configuration.
I hope that this upcoming release will attract a larger pool of prospective users, providing me with needed research participants.
!!!Reflections on Research Process
After many hours spent analyzing JW's interview transcript and log file, I asked myself: what is the value of my analysis efforts? Is the log file a trove of information or a rathole? I admit that I became distracted by the notion that my findings would have an impact on Overview's future design. It was around this time that JS revealed plans to remove major features and overhaul the user interface in the forthcoming release of the Overview web application; I believed that my in-depth analysis could validate or refute these design decisions. I lost focus on the larger goal of understanding how Overview was used in the process of exploratory data analysis.
This larger understanding of a process is not interface-specific (<<cite Beaudouin-Lafon2004>>); when we understand the process and its interactions, we can then evaluate specific interface components.
Ultimately, JS was fascinated by the detail and depth of my findings, but agreed that it was overkill for the purposes of validating design decisions or for identifying usability issues. What mattered most to him was that Overview had been used to write a story, and unexpectedly, that it had been used as a tool for streamlining an exhaustive search, rather than for its intended purpose, being a broad, sample-based variant of exploratory data analysis.
I have been rethinking my data collection and analysis methods, my methodology, and how my findings will eventually be presented to the research community. At the level of data collection, I have refined my interview foci to include a deeper examination of the difference between exploratory sampling and approximate search. More questions relating to information management and tag usage will also be added. At the level of methodology, the major question is whether to continue with a grounded theory approach and its constant comparative, bottom-up philosophy, or to instead survey a broad range of Overview users and then select specific case studies that appear to be radically different, given the users' goals. I could imagine reporting on cases of differing usage strategy (broad sampling vs. exhaustive approximate search), corpus provenance (records request vs. leak / dump), or user domain (journalism vs. digital humanities research). When presenting my findings, whichever approach taken will need to be consistent and well-justified.
!!!This Project in Context
A replicable evaluation of a visualization tool that supports the process of exploratory data analysis requires a methodology grounded in a generalized characterization of what this process is and what it isn't, an understanding of the form that this process takes across multiple domains. My current project is a small part of this dependency, in that it allowed me to study visualization use in a single domain. Over time, I will study exploratory data analysis and visualization use in other domains, both personally and via my ongoing comprehensive review of the literature.
I have already adopted multiple data collection and analysis methods; others will surely follow, subject to practical constraints and assessments of expected utility. A useful evaluation methodology and a mid-level task characterization are mutually dependent, and will develop together with further study.
!!Conclusion and Future Work
I am currently continuing my study and evaluation of Overview, a tool built for exploring large text document corpuses. I've still got a long way to go: in my first completed case study, I observed the tool being used for conducting an exhaustive approximate search; not exactly what I was expecting, but an interesting finding nonetheless.
I expect to interview more Overview users in the coming weeks and months. There is also likely to be an opportunity to elicit [[participation from journalism students|An Insight-Based Evaluation of Overview]] as they use Overview in the context of a course project. Such an opportunity would be logistically simpler than observing professional journalists, affording multi-session in-situ interview and observation methods (<<cite Chang2010>>, <<cite Saraiya2005b>>).
Along the way I've learned and reflected upon a great deal: methodological considerations, the process and pitfalls of mixed-method data collection and analysis, and my experience working with an external collaborator.
Writing this report has served to get my thinking directed toward my larger goal: the continued study of visualization evaluation and exploratory data analysis.
!Comments
''GC'' 12.09.29:
>"//thxs for sharing your report. I truly enjoyed reading it. very interesting work and I completely agree with your arguments (and methodology) for building mid-level, cross-domain task models and moving beyond time and errors. I did some preliminary work on that some years ago while developing an infovis for decision support, Valuecharts (you can check my three AVI papers if interested). I have actually gone back to that work recently in the context of two projects, one in which we may be able to test the infovis with real users (intervies etc.), the other in which we intend to perform experiments with eye-tracking to study user differences. The second reason why I am really interested in your work is the domain of the Overview system (text analysis) as well as the specific case study on an email corpus. As you probably know we have been working on text mining and summarization of emails, blogs etc. for quite some time. So I am really interested in the (new version of the) Overview system (for instance, in how it could be tailored to analyze emails/conversations corpora) and also in the specific Tulsa dataset. Finally, I believe your idea of working - at least in part - with students involved in a course projects is a good one, to collect more data and having more control on the experiments; in order to thoroughly explore different strategies, corpus provenance and different user domains.keep up with this great work!//"
''RR'' 12.09.30:
>''Clarifications, etc.:''
>
>The following (paraphrased) assertions are bold, and help motivate the thesis. However, they have the form of vague generalities. Need specific points, and concrete examples to flesh them out.
>
>//1. p.1 쵡ting user processes with and without viz can be highly informative.are these informative? What exactly do they typically give you? (at both formative and summative stages). Please give concrete examples.//
''MB'': Some examples:
*''Formative/without viz'': are users performing difficult cognitive / logical tasks that could be substituted by perceptual tasks? (<<cite Casner1991>>) - i.e. data lookup and value comparison, logical or arithmetic operations
*''Formative/with viz'': how / when is vis used? at what point in the data analysis cycle does it occur? what concurrent processes take place? what are the benefits and drawbacks of the current vis toolset (usability, utility)
*''Summative/without viz'': are users performing these difficult cognitive / logical tasks to a lesser extent? (<<cite Casner1991>>)
*''Summative/with viz'': how / when is vis used? how does it complement concurrent processes? usability, utility issues?
>//2. p.2 렴axonomies can become stifling if used for evaluation.쥡se give a concrete example of this. What exactly is the 欩ng�nt? Also, is this about particular taxonomies, or taxonomies in general?//
''MB'': high-level models have good descriptive power but low evaluative or generative power, low-level models have generative power but little in terms of descriptive or evaluative power(<<cite Beaudouin-Lafon2004>>); for example, <<cite Pirolli2005>>'s sensemaking task model can describe what users abstractly do in many domains, but it's difficult to equate one stage of the sensemaking loop in one tool/domain with another tool/domain. Conversely, low-level interactions like <<cite Amar2005>> or <<cite Casner1991>> reveal a chain of interactions, which doesn't reveal a purpose in itself; these chains of activity must be grouped and linked to abstract higher-level tasks, and at that point we can make the comparison between two domain/tool pairs.
>//3. p.3 Spell out a bit more what grounded theory is, and how it's helped. Please give a concrete example of this. Especially good would be an example of something that grounded theory can do that nothing else can.//
''MB'': We're using GT because we're indirectly studying a process. We need to reconstruct a user's process from the retrospective story they tell us, we don't initially have our own observations to rely upon. This is an intersubjective construction that relies on the medium of language, not on our own subjective construction via the medium of observation. While we have preexisting theoretical concepts of what the high-level tasks are, we can't assume that journalists operate in the same way as academic researchers, intelligence analysts, etc. We also don't trust these models, which is why we need better task taxonomies. It's a cautionary approach, these preexisting theories can guide our questions, but we're not going to categorize activity at the outset.
There's also some examples in the literature of studying visualization use in workplaces and collaborative visual analysis among teams of domain workers (<<cite Isenberg2008>>, <<cite Tory2008>>) that couldn't easily be studied with a strictly observational approach.
>//4. p.1 Two different kinds of extensions to evalution are mentioned: a. what is measured (metric); b. the context in which it's measured. Give concrete examples of each, showing they transcend traditional approaches.//
>//(Also: In the work here, it was discovered that the Overview system sped things up. Couldn't this be seen as just an application of the traditional metric of time? Not clear how the new stuff helps.//
''MB'': many new evaluation methodologies attempt to quantify / qualify insight (<<cite Saraiya2004>>, <<cite Saraiya2006>>, <<cite North2011>>), hypotheses generated / hypotheses validated (<<cite Jianu2012>>), problems solved (<<cite Mayr2010>>), types of learning: about the data, about the analytical task, about the tool/vis. technique (<<cite Chang2010>>), understanding strategy (<<cite Ziemkiewicz2012>>). In our case, we have reported a qualitative account of a particular strategy, about how a hypothesis was validated. As a byproduct of this, we also reported the time required to do the analysis, and the volume of data processed during that time.
Re: context, I refer to outside of controlled laboratory settings, in the context where analysis takes place on a day-to-day basis. In our case, this was a newsroom. In the literature, contexts have varied from the research labs of domain scientists (<<cite Saraiya2006>>, <<cite Ziemkiewicz2012>>), classrooms of intelligence analyst trainees (<<cite Kang2011>>).
>''Methodological foundations/assumptions:''
>
>//5. Focus on successful users alone creates a baseline bias w whether the system helps in general. (If you find someone who says that their favorite religious being helped them with something, should we believe this is generally true? What about the people who tried it but didn't get any assistance? Shoudn't that be taken into account?) How would you overcome this problem?//
''MB'': this is an artifact of how users are recruited, and the nature of domain users' work, a problem which goes beyond this study. For a web-deployed desktop tool such as Overview, we don't know who is using it and we likely don't hear from the population of users for whom Overview doesn't meet their needs, because they choose not to contact us. This could improve with the web-based version of the tool, since we have logged user login data, we have email addresses, we can survey users. Second, the successful users tend to be grateful, allowing us a couple hours out of their busy schedule to tell us about their process. Unsuccessful users may not be willing to give us their time because they haven'y yet received the payoff.
''MB'': (post discussion): be aware of retrospective recall biases, misattributed reasons for success and misattributed reasons for failure
>//6. p. 1 stated that the goal is to understand exploration (to discover and understand), rather than lookup and verify. However, the case study was about verification. Was this really all that relevant?//
''MB'': Looking back, I'd describe the case study as a process to "discover and verify", not "lookup and verify"; JW still required exploration. As we discussed, exploration and search is not a dichotomy but a spectrum. This broad categorization at the outset of the paper was premature.
Also, I admit a proclivity towards "discover and understand" because that is what JS built Overview to support.
>//7. More generally, what if journalism is mostly verification? Would it still be a good domain to study exploration? What would you do?//
''MB'': This is why I think it's interesting to study multiple domains who use Overview: digital humanities, communications scholars, historians, social scientists. There may be different strategies across these domains. I also return to the point that exploration and verification form a spectrum, not a dichotomy - we should expect some variation among journalists.
>//8. When you measure performance, how do you know that you're getting performance under normal conditions? (cf. Hawthorne effect - the effectiveness of a tool depends largely on the amount of mindful attention being devoted in its use). How can you really say that some system helps when it췡ys the case that people do better when they know they�itored?//
''MB'': This is the case in both laboratory and real-world settings, and not a problem isolated to this research. It's a limitation that we're aware of. However, the web-based version of Overview will log interactions, a process transparent to users. If there is a Hawthorne effect perhaps we'll see it there, when comparing users we talk to to those we don't.
>''Nature of exploration process:''
>
>//9. Insight and serendipitous discovery might rely on pre-existing concepts lurking around at some level. Is it true that it's possible to have absolutely no questions/preconceptions in mind when exploring? Or maybe just simpler ones are used? (cf. debate in philosophy of science as to whether 彡 ever exist.) If there are pre-existing concepts, what are these? How could you find out?//
''MB'': Some existing taxonomies take intent-free exploratory browsing into account - just to see what's in a dataset (i.e. <<cite Mullins1993>>). In interviews, I try to tease out the user's initial questions or goals: what was their ideal goal state? Finding a trend? Verifying a hypothesis? Forming a hypothesis?
Re: serendipity and insight, <<cite Andre2009>> and <<cite Klahr1999>> explain that for serendipitous discovery to occur, one requires a prepared mind. By prepared, this could mean a wealth of prior domain knowledge, an understanding of suitable analogies and metaphors, initial hypotheses.
>//10. How much needs to be known (or not known) ahead of time for what kind of task? Could there be a continuum between exploration and search? At what point does the problem become ill-defined? //
>>//箬 looking for a bird; know if when you see it. Is this search or exploration? How about looking for an animal? Looking for a pattern? Looking for a solution? Granularity alone may not be relevant, but level of structure/category.//
>>//ࡠcontinuum, what are the relevant distinctions/stages/levels? Do these require different kinds of process?//
''MB'': Exploration and search is a continuum, and this is what we're trying to taxonomize / categorize for visualization; same for degrees of ill-definededness. <<cite Marchionini2006>> is helpful for calling out the distinctions between these activities. In our case study, lookup would allow him to call up an email with a known sender/subject/recipient. Browse would allow him to look up emails matching known sender/recipient or sender/subject or recipient/subject pairs. But perhaps he only a known subject, with many senders and recipients. Or worse, he has date range, or keywords appearing in email body texts. What does he do then? Each requires some varying shades of exploratory activity.
>//11. Another possible dimension of exploration: exhaustive search vs sampling. The former involves individual items (to e.g., verify/refute) vs statistical patterns. In both, could proceed until some criterion is reached (a level of confidence). But in the former, the key information might have measure zero 㴠an item or two among zillions. A single counterexample can disprove a general statement; a single piece of evidence can sometimes prove who did what. Very different than looking for e.g., whether there exists a general bias, which would be a population/statistical effect.//
''MB'': Agree completely. In JW's case, his key evidence was scattered about ~120 emails out of 6,000.
>//12. Instead of validating/falsifying hypotheses (p. 3), might it be better to pursue multiple competing hypotheses? (cf ACH Heuer) Perhaps evaluate in terms of whether this process is supported?//
''MB'': I address this in interviews, where I refer to multiple hypotheses: I ask not only about how many hypotheses were validated, but also how many were generated over the course of one's analysis process.
>//13. Perhaps check out literature on exploration processes in related domains? (e.g. scientific discovery). Process may be similar.//
''MB'': Yes. I've consulted <<cite Springmeyer1992>>, <<cite Klahr1999>>, information retrieval and library sciences research.
>''Miscellaneous points:''
>
>//14. What particular aspects of Overview enabled JW to speed up his search? Was there a change in the process, or simply the same process done faster? How would you know? (cf earlier comments about metrics )//
''MB'': (1) personal information management: his tagging and categorization scheme, (2) the systematic and depth-first approach to reading/skimming clusters of related documents. The clustering Overview did a priori made it easier to dismiss or easily tag documents en masse, whereas had the documents been presented as a flat list, this would have involved context switching, longer comparison times, and more time spent reading each document.
>//15. p.4. (bottom) compliment -> complement (yes, there really is a difference...)//
''MB'': Yes. TM has corrected me before for this error.
>//16. p.5. What exactly is a "botched ... equipment purchase"? Reads oddly栉 botch a purchase, what does that mean?//
''MB'': Public opinion on the matter was that the purchase of police equipment was badly handled: over-budget and outdated equipment that performed poorly, didn't integrate with existing equipment, and were disliked by users. The shadier aspects of the purchase revolved around corporate kickbacks to municipal and police officials handling the purchase
>//17. For more info on grounded theory - contact Brian Fisher//
''MB'': Will do.
>//18. To get hold of conveniently-located journalism students - contact UBC School of Journalism//
''MB'': I have a contact there who's aware of Overview. I contacted some VIVA folks back in May but never heard back. The UBC VA challenge group currently had no ties to UBC journalism (as of July).
!Meeting Minutes: ~TM-RR-12.10.03
*talk to B. Fisher (SFU SIAT), ~PhD student Richard Hernandez re: [[Grounded Theory]]
*on individual differences:
**low-level biases and individual differences in perception may propagate up to levels of cognition, pattern-recognition, trend identification, EDA
*contact UBC journalism
**to do: write pitch to UBC journalism, abbreviated version of this report's intent - send to Ron prior to ~VisWeek
***could also be sent to UBS humanities
**paired analysis (citation?)
*a note on "time-out follow up" - what does this refer to?
*mid-level task taxonomy ~ domain-isomorphic tasks
*to read:
**<<cite Rensink2012>> - distinguishing perspecitves, foundations of a new science
**Heuer's //Psychology of Intelligence Analysis// (in [[References: To Read]])
**Beveridge's //The Art of Scientific Investigation// (in [[References: To Read]])
**CP Snow's //The Two Cultures// (in [[References: To Read]])
**Duhem, P. [[The Aim and Structure of Physical Theory|http://www.amazon.com/Structure-Physical-Princeton-Science-Library/dp/069102524X/ref=sr_1_1?s=books&ie=UTF8&qid=1349395994&sr=1-1&keywords=duhem]]
>//"Nice treatment of how observations can be theory-laden."//
**Kuhn, T.S. (1970). [[The Structure of Scientific Revolutions, 2nd ed|http://www.amazon.com/Structure-Scientific-Revolutions-50th-Anniversary/dp/0226458121/ref=sr_1_1?s=books&ie=UTF8&qid=1349395909&sr=1-1&keywords=kuhn+structure+of+scientific+revolutions]]. Chicago: University of Chicago Press. (in [[References: To Read]], purchased Oct. 6, 2012)
>//"Famous book about how scientific understanding = creating a paradigm."//
**Howard Margolis' [[Paradigms and Barriers: How Habits of Mind Govern Scientific Beliefs|http://www.amazon.com/Paradigms-Barriers-Habits-Scientific-Beliefs/dp/0226505227]]
>//"Interesting expansion of Kuhn's points, with close ties to the development of theories/concepts."//
**Popper, K. [[The Logic of Scientific Discovery|http://www.amazon.com/Logic-Scientific-Discovery-Routledge-Classics/dp/0415278449/ref=sr_1_1?s=books&ie=UTF8&qid=1349395946&sr=1-1&keywords=popper+karl]]
>//"Another classic about the discovery process. Emphasis is on ability to test a theory, so that it can be rejected."//
**Lakatos, I. (1978). [[The Methodology of Scientific Research Programmes: Philosophical Papers, Volume 1|http://www.amazon.ca/Methodology-Scientific-Research-Programmes-Philosophical/dp/0521280311/ref=wl_it_dp_o_pdT1_nS_nC?ie=UTF8&colid=EMOS2FV76X1D&coliid=I3RDVQHOJLCV6A]]. Cambridge University Press.
>//"A more interesting (IMHO) treatment of how discoveries are made. Makes the important point that it's impossible to either confirm or reject any theory/hypothesis; it's overall consistency that counts."//
**Kantorovich, A. [[Scientific Discovery: Logic and Tinkering|http://www.amazon.com/Scientific-Discovery-Tinkering-Philosophy-Biology/dp/0791414787/ref=sr_1_5?s=books&ie=UTF8&qid=1349396209&sr=1-5&keywords=tinkering+science]]
>//"An interesting take on discovery as tinkering with existing knowledge, rather than creation de novo. Good case for incremental view of exploration/discovery."//
!References
<<bibliography>>)
Adaptive Control of Thought, a cognitive architecture developed by Anderson et. al., assuming two kinds of knowledge: declarative and procedural.
[[ACT-R|ACT]], the most recent model, includes a goal memory and IF/THEN production rules for comparing what is held in declarative memory to goal statements.
The [[ACT-Scent|ACT]] architecture and [[SNIF-ACT|ACT]] models, developed by Pirolli et. al., describes information foraging as an extension of the ACT model, containing an information scent module which evaluates the utility of production rules for comparing goals with chunks of declarative knowledge. Spreading activation occurs in the declarative and information scent modules.
Source:
<<cite Pirolli2009 bibliography:Bibliography>> ch. 1 p.9, 25
[>img(33%, )[Comic from the New Yorker|http://2.bp.blogspot.com/_3DsAgfOGgvI/SeTMyQKeJPI/AAAAAAAAAR4/klmnpjWRdy0/s400/090420_cartoon_b_academia.gif]]
Here are my notes from the [[Graduate Pathways to Success (GPS) Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]'s workshop sessions (Oct 19-20, 2011), presented by [[Simon Clews|http://www.simonclews.com/]], Writing Centre for Scholars and Researchers, Melbourne School of Graduate Research, University of Melbourne.
While these notes won't be nearly as valuable as attending the workshop itself (and participating in the writing exercises), they may be somewhat helpful if you plan on ever attempting to write about your expertise in your field to a non-academic audience. I really recommend attending the workshop, should it be offered again in the future.
!The take-home message
Above all else, the following ''mantra'' will help you write, and in particular it will help you write commercially (that is, writing for non-academics, and profiting from it):
#''Read more'': read within and outside of your field, keep up to date with what's going on in the world, in your country, in your larger field (i.e. computer science), in your specific field (i.e. HCI). Read newspapers, blogs, [[Google News Alerts|http://www.google.com/alerts]] for your field. Read both well-written and poorly-written, as long as you always read critically. Read the //New Yorker// for examples of good creative non-fiction.
#''Write more'': write about your research, write about anything, and write often. Write for a specific reader (i.e. your mom), and/or write for an imagined reader or stereotypical reader (i.e. pick a demographic to write for), write for kids.
#''Edit more'': edit mercilessly, read your work out loud, edit top-down and bottom-up, edit back-to-front and front-to-back, edit with multiple passes. Always have a red pen handy. Plan to edit for as long as you plan to write.
This mantra is the most important take-home message. If you stop reading here, you'll already be well on your way to better writing. The following will also be helpful, but not as critical.
!Why do it?
Why should you write for the general reader? For non-academics?
*The general reader is ''interested''. Creative non-fiction, popular science, and humanities articles/magazines/books are very popular. They want to read about you and your research (this is unlike students and researchers who read your academic writing because they need to learn, or they need to stay current/informed of progress in the field, or they're reviewing your paper)
*They're ''not out to get you'' (unlike some academic paper reviewers).
*They ''trust you''. You're a scientist! An expert!
*They're interested in you as a person!
*Odds are you'll have a ''wider readership'' writing non-academically.
*If you're research is funded by taxpayer dollars, you have an ''obligation'' to write to taxpayers.
*Writing's a ''career-smart'' move. A good writer may attain financial gain, but it's more likely that you'll have lifestyle gains (people will seek your expertise, invite you for talks, dinners, conferences). You'll also learn how to manage as a small business.
!On writing
Some ''golden rules'' when writing for the general reader:
*Abandon footnotes. Readers don't want to read the literature review your thesis/dissertation.
*Step back from your expertise. Assume zero knowledge. You've mastered a field, good work; show off your mastery in your thesis. When you write for the general public, you must imagine reading as a casual reader. What will they need to know to understand your work? Qualify unknown concepts Give context: introduce important people, places, acronyms and concept - these may be well known in your field, but you must assume zero knowledge from the general reader. Example: "//Bill Buxton from MSR gave a presentation on sketching ~UIs at CHI in 2011//" - we in the field know what this means, but a general reader would ask "Who's this guy and why's he important? What's MSR? What's CHI? sketching? ~UIs? why is the date important?"
*Don't use academic words. "Discourse, Argument, Candidate" mean different things (or don't mean anything) in the "real world".
*''Tell a story''. The brain is hard-wired to feel engaged when hearing/reading a narrative.
*Put yourself into the story (as much as possible). Non-academic readers are interested in ''you'' and your connection to the story.
*Be conscious of your reader and never talk down to them (remember to step back from your expertise)
*''Think short'': short words, short sentences, short paragraphs (read Ernest Hemingway for a good example of this).
*Use ''active verbs'' over passive verbs. The latter is academic. Replace "A study was conducted" with "I conducted a study".
*Don't use technical jargon or acronyms (or use sparingly)
*Don't use filler words unless necessary. If using qualifying adverbs/adjectives, emphasize them by using them seldomly. Mark Twain was known for substituting "damn" for every time he would write "very", much to his editor's dismay.
*Read George Orwell's 1946 essay "[[Politics and the English Language|http://blogs.ubc.ca/rmst221/files/2010/01/orwell-pol-en.pdf]]" for more writing tips. Strunk & White's "[[The Elements of Style|http://www.amazon.ca/Elements-Style-William-Strunk/dp/020530902X]]" is also a handy pocket guide.
!!Writing online
[>img(50%, )[F-shape patter for reading web content|http://www.useit.com/alertbox/f_reading_pattern_eyetracking.jpg]]
Be aware that ''most online writing is bad''. Most of it never gets read. And it stays there (this permanence can be both good and bad). Regardless, consider the following:
*Who is your ''audience''? What do they want? Where are they coming from: search engines, incoming links, social media?
**''Reading the web is non-linear''. Expect a reader to start at any page on your site/blog.
*Have a mix of ''active and passive'' media. The web is unlike a library or bookstore, stays on web pages are very brief.
**Expect less audience commitment (relative to when one is engaged reading a print book)
**Be aware of eye-tracking study results that show the F-shape pattern for scanning web content. Don't place important material in the lower right corner.
**People scan, but when they stop to read, they read slower than they would for print material.
*Every page should have a ''purpose'' and a reassuring reminder or brief introduction about the site (i.e. a sidebar with your name and 1-2 line bio).
*Links should be ''logical and useful''.
*Like newspaper articles, write with an ''inverted pyramid'' in mind (see image). Remember readers will stop reading at any time. Be concise, clear, and true.
[>img(25%, )[inverted pyramid of writing|http://www.youbrandinc.com/wp-content/uploads/2011/07/InvertedPyramidGIF.gif]]
!!Writing letters to the editor
You won't get paid for these, but you should consider writing to the editor of your newspaper/magazine of choice, whenever a news item related to your field of interest comes up.
*Have a beginning, middle, end.
*Avoid motherhood statements, such as those beginning with "Would you...?"
*Keep a 1st person perspective, use active verbs.
*Acknowledge sources (i.e. previously published letters, articles, publications)
*Try not to get sued.
!!Writing articles & opinion pieces for newspapers/magazines
You might get paid for these. Usually these come after you've written some well-received letters to the editor or reviews. But perhaps your expertise is one in which you're able to start with an article.
!!!The pitch
First you need to write a pitch for the article (a.k.a. story):
*Determine which publication you're going to pitch to. Read the publication to ensure that you're pitching to the write place. This is market research. If they've recently published several articles in your field, it's not wise to pitch to them now (it shows you don't read the publication).
*Determine who your reader is.
*Determine where your story belongs in the publication. Who is the reader?
*Contact the right person. Do your homework. The editor will throw out your pitch if it says Dear Sir/Madam.
*Write a convincing pitch. Answer these questions:
**''Why this''? why is the subject matter important? who will think it's important?
**''Why now''? are there current events that make the subject matter relevant?
**''Why you''? what is your personal connection? your expertise?
!!!The article
*produce good clean copy (no need for edits, the write word length), on deadline.
*the perfect story has:
**a witty title
**a hook
**a stunning first paragraph
**a circular structure
**a good quote
**a good anecdote
**a question and an answer (no unanswered questions)
**a startling statistic
**a killer closing line
*if you're writing an opinion article, acknowledge contrary opinions but maintain that yours is superior (unlike academic articles or philosophy essays)
!!Writing books
Not every thesis has a book in it. It's more likely that your research process has a story in it. So rather than fruitlessly attempt to use your thesis as source material, start from scratch.
*What parts of your body of research would be interesting to the general reader?
*Address these questions before you begin:
**What sort of book is it? Creative nonfiction? Popular science? Memoir? Autobiography? A book for children / young adults?
**Who is the audience?
**What is the style?
*Avoid the ''slush pile'' at publishers' desks (actually, many publishers don't accept unsolicited manuscripts). If they do, they're read by junior interns ("poison tasters"). Find the write publisher and establish personal contact.
*Before you seek out publishers, go and "look" for your book in a bookstore. What section would it be in? Ask an attendant to help you "find" your book (ask for books on similar subject matter). Where would your book sit on a shelf (your last name might matter here). Identify agents, editors, publishers. "Look" for your book on Amazon too, look at other recommended books.
*Do you ''need an agent''? Depends. Initially, they're harder to find and pitch to than publishers. Get some initial success under your belt first.
!!!The book proposal
A good book proposal has:
*a title page and table of contents
*a summary / dust jacket blurb about the book. 3rd person, racy humourous.
*an "about the author" statement. Don't be cryptic.
*answers who the audience will be.
*lists the competition. Similar works, publishers, authors.
*mentions opportunities for publicity / promotional activities: events, etc.
*an outline (more detailed than Table of Contents)
*2 sample chapters
*is well presented and formatted.
!!Writing book reviews
Write about books in your field. Find out who publishes books in your field and get on their mailing lists for advance copies by citing your interest and expertise. Pitch your review to publications or newspaper literary/book columns. They usually have a need for material here - their columnists don't have time to read everything. When writing the review:
*Provide context, content (the "no spoilers" can be violated with non-fiction), and comment.
*Give a description of what the reading experience will be like.
!!Writing for kids
"If you can't explain yourself to a six year-old, you don't understand it well yourself." - Einstein
Writing non-fiction for kids is a wildly successful area. What you should do:
*Think like a kid (know your market). Hang out with kids (if possible), go with them to bookstores and help them "find" your book or magazine.
*Play with words. Kids love this (I do too).
*Be conversational.
*Play with unusual formats.
*Link new information to something kids already know
*Include or suggest activities they could do alone, with their friends, with parents, or with schoolmates
*Tell a story
*Use and qualify reliable sources
*Use photo research
*Contact publishers of kids' books (they specialize with kids books and don't shy away from unusual formats, lots of images)
!!Writing for museums / exhibitions
Yes, these need writers with expertise as well. Some things to keep in mind:
*Like the web, people attending a museum/exhibition won't proceed linearly through the exhibits. They can start anywhere and go to anywhere. They will also skim. Recall the inverted pyramid and ensure that each exhibit has enough self-contained information to be of value.
*Provide a big picture - what are the big take-home ideas?
*Understand who your audience is. Kids? Adults? Mixed?
!!Other mediums
There are other places where there is a need for non-academic writing, for general audiences (although some venues are highly unlikely for most of us):
*Documentary films
*Amusement parks (i.e. NASA rides at Universal Studios)
*Traveling attractions (i.e. Walking with Dinosaurs)
!On editing
We write too much. Approximately 40% too much. Here are some editing tips:
*Edit for as much time as it takes you to write.
*Step back from the work before editing (within reason). Take a walk, have a coffee. Better yet, finish writing before the weekend and start editing the following week.
*Edit when you've finished writing, not while writing. You can't wear your editing and writing hats at the same time.
*Expect many drafts.
*Be aware / on the lookout for your bad habits.
*Don't expect the luxury of an editor / a second set of eyes to read over your work.
*How to edit:
**the best sentence bypass: find the best-written [sentence/paragraph] in your work. If your work holds up without it (i.e. if it doesn't advance your argument), ''cut it out'' (it may be hard to adopt this practice at first, but apparently it works). Do this recursively until you can't possibly cut any more.
**edit back-to-front, front-to-back.
**edit multiple times:
***1st pass: no red pen, comfy arm chair. Imagine reading as a specific reader.
***2nd pass: enter red pen. Make suggestions, comments, underline.
***3rd pass: make structural / high-level changes
***4th pass: individual line edits. Check spelling, capitalization, punctuation, formatting, word usage, continuity, grammar, sentence structure, subject/verb agreement, and verb tense, replace passive verbs with active verbs, remove or qualifying jargon, cliches, and acronyms.
***5th pass: no red pen, comfy arm chair, read the result. Imagine reading as a specific reader.
!Activities
These ''activities'' will help you write better:
*Maintain a blog, website, twitter, facebook, etc. Acknowledge that most writing on the web is bad writing (and most of it hardly read), but write nevertheless.
*Give talks inside and outside of academia. Give talks to groups in other fields, give talks to community groups (i.e. historical societies, retirement communities)
*Write letters to the editor (one of the easier ways to get your name in print). If your letters are well-received, work your way up to opinion pieces pitched to or commissioned by the publication. This is the path to short and long articles.
*Call in for radio talk-back, regardless of whether your field of research is related to the topic being discussed. Whenever you have an opinion to give. If the topic it is related to your field, calling in and citing your expertise could land you on the radio show's list of "go-to" people of people working in the field.
*Write book reviews (another easy way to get your name in print). Ask for advance copies from the publisher when a book in your field is due for release (cite your expertise). Pitch a review to relevant publications or literary supplements of local/national papers.
!Practicalities
Writing commercially means operating as a small business. You are selling a service. Here are some practical and professional tips that will help you stay prepared, professional, and sane:
*Take the necessary steps to operate as a small businesses (tax claims, business expenditures, etc.). Have a good accountant
*Have a copy of the [[latest writers' directory for your country|http://www.amazon.ca/Canadian-Writers-Market-17th/dp/0771085281/ref=sr_1_5?ie=UTF8&qid=1319754626&sr=8-5]] - where to sell your articles. Each country has a similar guide, published annually.
*Use a proper, professional email address that you've paid for (no gmail/hotmail)
*As a writer, you must have a platform: Maintain a blog/twitter/facebook/etc.
*Maintain a database. Keep good records of where you've sold articles / pieces.
*Produce professional invoices when selling articles.
*Keep a clean, tidy desk in a light and airy room
*Write every day.
*Keep in mind that you also won't be wealthy by writing alone, especially at first. You'll need another means of income (a second job, a supporting partner).
*Be optimistic, have a thick skin. You will get turned down often. You're in it for the long haul.
*Be strategic and think creatively.
!Exercises
At the workshop, we ran through several fun and challenging writing/thinking exercises:
*Introduce yourself and your research to an older, well-educated, white-collar man. Then, re-introduce yourself and your research to a 10-year-old boy.
*Write a book proposal based on your thesis research.
*Write the first paragraph of "the perfect story", a magazine/newspaper piece relating to your research. Identify the specific publication and type of reader.
*Write a pitch for your "perfect story".
*Present your thesis and frame it within your biography, extract stories from both.
*Attend/participate in "3-minute thesis" competitions
Referring to in-depth participatory qualitative field research, which aims to find ways to change or improve processes over and above the pure anthropological or sociological [[ethnographic approach|Ethnography]] (studying processes for their own sake). (Lewin C. (Ed.) (2004)
*[[EPSE 595]]: [[Action Research|http://prezi.com/0lc7aiia6rlx/action-research/]]
!![Hayes 2011]
<<cite Hayes2011 bibliography:Bibliography>>
TM's notes:@@color:#444bbb;
>''yes'', [this describes us], we definitely do this:
*action as explicit part of the process of inquiry (learning through action)
*scientist's personal involvement/role as essential not 'bias'
*don't privilege generalizability, instead transferability
*skilled at opening up lines of communication not distant objective observers
>''some'', [this describes us to some extent], we sort of do this (but probably not as much as they do):
*goal of learn through action to get through design paralysis (messy/huge space)
**//true that we have a huge design space, but i haven't seen paralysis as a real pitfall//
*prolonged engagement
**//but maybe by their standards we're not so long...//
*triangulation for credibility
**//we're not nearly so concerned about gathering all potentially opposing viewpoints//
>''no'', [this does't describes us], explicitly opposite from our methods/arguments:
*participant language and perspectives as opposed to layering of scientific language from the literature on participants concepts
**//we are explicitly advocating exactly the translation of participant concepts into our language!//
*researcher as facilitator not leader, avoid 'model monopoly'
**//we are explicitly saying that it's the vis researcher who must lead on designing solutions because they know the vis design space and the user does not. this stance is fundamentally opposite the AR party line!//
*participants as full partners in writing process
**//for us, relatively rare and not something we have as an active goal//
>''no'', [this isn't even relevant], totally orthogonal to what we care about:
*all the political stuff
**//socially relevant, inherent value to society, sustainable social change, democratic/inclusive, traditional research as authoritarian, "solidarity with oppressed... adversarial role to the powers that be"//
*most of the theoretical stuff
**fundamentally postmodern,
>so - i would be very careful to not say that what we do is "closest" to action research. there are several points of divergence, some about stuff they care about that we don't, but worse yet some where our stance opposes theirs.
>
>I do think it's safe to say that some of their methods and arguments resonate strongly with us, as long as we're very clear about where we diverge. for me, that's obviously all the political/pomo stuff, but somewhat more subtly this issue of scientific authority - a mantle that we do not reject.@@
Rich data in the form of notes, pictures, artifacts from a qualitative study (i.e. a [[Contextual Inquiry]]) or a [[Observation Study]] are posted on a wall and iteratively grouped into a hierarchy of related themes. Themes are not predetermined, but arise out of the data, as in [[Grounded Evaluation]].
This project will form the basis of part of my RPE, a 4 month research stint, complementing a parallel [[interview study|Document mining in data journalism]] of journalists using Overview to perform document mining.
!!Introduction
In the fall of 2010, the [[WikiLeaks organization|http://en.wikipedia.org/wiki/WikiLeaks]] released [[nearly 400,000 text documents|http://goo.gl/TAgOs]] relating to the conduct of US armed forces and independent security contractors during the war in Iraq. Since that time, specialized investigative ᠪournalists�reported on what was ''discovered'' in this vast deposit of documents, which included startling information regarding civilian casualties, friendly fire incidents, and observed breaches of protocol. This proposed [[Insight-Based Evaluation]], in brief, asks how data journalists 彨 deposits using alternative data analysis methods (with and without Overview), and how they come to discover unanticipated yet newsworthy stories hiding in the data.
!!Research questions
What constitutes insight, or a unit of discovery? When is insight the means (hypothesis generation) or the ends (hypothesis validation)? Can we establish a shared interpretation of insight that traverses domains? Until this time, the use of an [[Insight-Based Evaluation]] to study the effectiveness or utility of a visualization application has occurred within the context of applications for use in life sciences domains. This project is also intended to deploy this evaluation method to another domain, namely journalism, for the first time.
!!Related work
The original [[Insight-Based Evaluation]] paper <<cite Saraiya2004 bibliography:Bibliography>>, published in the proceedings of ~InfoVis 2004, with an expanded journal paper version the following year (<<cite Saraiya2005b>>), described a controlled laboratory study aimed at quantifying insights as a means of evaluating several visualization alternatives in the bioinformatics domain. They recruited life sciences students for a between-subjects study and instructed them to explore a dataset, reporting what they discovered using the [[Think Aloud Protocol]]. Observed metrics included the time to arrive at first insight, the number of insights-per-minute, total number of insights, the total time spent exploring the dataset, as well as metrics assessed by a representative domain expert: categories of insights (summary, breadth vs. depth), the correctness of insights, and the domain significance of insights. A similar methodology was used in a recent study by <<cite O'Brien2010>>, also within the bioinformatics domain.
By 2006, the Saraiya et. al. group was considering alternative approaches to the controlled laboratory experiment. A position paper by <<cite North2006>> suggested a longitudinal approach, a qualitative comparison of insight categories. This was executed in <<cite Saraiya2006>> with 2 professional bioniformaticians reporting their insights during the course of their regular ongoing work in a diary/journal format, with insights being more significant and deep as the period of analysis went on. This qualitative re-characterization of insight was echoed by position paper that followed by <<cite Chang2009>>.
In 2010, the Saraiya et. al. group returned to the laboratory experiment, this time comparing the benchmark task method with the insight method (<<cite Saraiya2010>>, <<cite North2011>>). Despite collecting quantitative results as evidence to suggest that the insight-based method accounts for all results generated by the task-based method, they found that their qualitative results were more usefully informative. Quantifying the significance of insights revealed little variance, most were surface-level details. This suggests a shift away from attempting to quantify insights generated by participants, or relying on domain experts to rank or grade insights, and towards a researcher-led qualitative open-coding of insights into conceptual categories. These categories are validated at a later stage of analysis by a domain expert. The appeal of this approach is that it will likely require less involvement of a busy domain expert, offloading most of the analysis to the researchers, and would allow for a more qualitative comparison between the alternative visualizations.
!!Methodology
This section describes a longitudinal insight study comparing document mining with and without Overview.
''Between or Within Subjects?'': a between-subjects study would entail a single dataset and one half of participants using Overview to analyse it, while the other half of the participants use other existing means. Alternatively, a within-subjects study would require a second dataset, with all subjects experiencing both conditions (with Overview, without Overview). Participants would require some familiarization with Overview and other existing document mining techniques.We'll need to determine how much time data journalism students can afford to give us. This may depend on the level of involvement of faculty, and whether working with us could serve double-duty for a course assignment. I'm currently writing a iterating on a proposed methodology for this insight project, part of which will be a pitch for journalism faculty. Do you have anyone particular in mind for this?
The ideal would be a longitudinal within-subjects study: we provide participants with 2 datasets of similar size and complexity (A and B), 2 protocols (with Overview, search only), and an equal amount of time for each protocol (several days / weeks). For example:
*Participant #1 - Overview (A) first, search-only (B) second
*Participant #2 - search-only (A) first, Overview (B) second
*Participant #3 - Overview (B) first, search-only (A) second
*Participant #4 - search-only (B) first, Overview (A) second
*etc.
''Longitudinal ~In-The-Wild vs. ~Single-Session ~In-The-Lab?'': Should we be able to recruit local students (i.e. those from the UBC school of journalism), we may be able to conduct controlled laboratory observation sessions. This could provide an opportunity to triangulate on findings emerging from the longitudinal approach. However, this approach would require a lengthy session, which would include a demo and time spent familiarizing oneself with Overview. It would also likely necessitate a smaller toy dataset, as the time required to explore even a modestly-sized collection of documents is in the order of hours or days.
For this reason, a realistically large and complex dataset, in a longitudinal setting may be more appropriate, with a study period conducive to the academic calendar of student participants and faculty members collaborators (on the order of weeks or at most a couple of months). Even with the potential for local students, the methodology should remain longitudinal. An exit interview, a show-and-tell session (over Skype screen sharing or in person) will occur in lieu of an observational laboratory session involving the [[Think Aloud Protocol]].
!!!Participants
''Students'': we will require participants to have sufficient background domain knowledge in the field of journalism. While a longitudinal [[Insight-Based Evaluation]] may be conducive to domain expert professionals and their normal existing workflows, a comparison of analysis tool alternatives and an imposed diary recording protocol may not be appropriate given their tight deadlines. As a compromise, we can recruit journalism students enrolled in professional graduate programs in data journalism, individuals with both sufficient background domain knowledge and time to participate in a study. Being in the data journalism specialization will also (presumably) ensure a sufficient technical background. This compromise has been taken by previous insight studies, with the exception of <<cite Saraiya2006>>, the longitudinal study of 2 professional bioinformaticians. A drawback to using students is lack of motivation or incentive; the participants are not engaged or interested in the dataset to the same degree as that of professionals engaging with a dataset during the course of their ongoing work. Offering monetary compensation for participating in the study is one way to motivate participants. Alternatively, another possible solution would be to enlist faculty and instructors affiliated with these data journalism graduate programs, combining this study with their course curriculum such that students receive course credit for participating. Participation in this study could culminate in the generation of a course assignment.
The number of students recruited may depend on the level of coordination with faculty, the time permitted to us, and whether we conduct a between- or within- subjects study. <<cite Saraiya2004>> report a between-subjects study with 30 student participants, 6 visualization tools, and 2 datasets. Each user was assigned one tool and one dataset. By comparison, the <<cite O'Brien2010>> study used a within-subjects protocol, 5 participants, and 2 visualizations. The longitudinal method of <<cite Saraiya2006>> permitted their 2 domain expert participants to use the suite of visualization and analysis tools that they would use during the course of their normal, ongoing work. The most recent was a 60-participant between-subjects study (<<cite Saraiya2010>>, <<cite North2011>>), with 10 users for each of the 3 visualization alternative for 2 evaluation methods (task-based, insight-based). Since we we will likely be adopting a longitudinal, qualitative approach, we should keep the number of subjects small. A between-subjects study, comparing document mining with and without Overview, would double the number of subjects required for the study. Alternatively, a within-subjects study would require double the time to complete the study, and a second dataset (one for condition A (with Overview), another for condition B (without Overview)).
''Faculty / Domain Experts'': we will require the assistance and cooperation of a data journalism faculty member or instructor. This level of corporation could be restricted to allowing access to students, advertising the study, and/or offering course credit for participating in the study. Requiring participation in the study for completion of a course assignment, such as a written article or story, would be ideal. While we the experimenters would collect, code, and categorize insights generated by participants, we would hope to enlist faculty members to verify / critique our categories of insights. In order to make these assessments, or to quantify the domain significance of insights, these faculty members would require a high degree of familiarity with the datasets used in the study.
!!!Data collection methods
A longitudinal approach precludes observational methods and the [[Think Aloud Protocol]]. However, if participants are accustomed to taking voice memos during the course of their analysis, we can collect these. Our primary source of data will be diary entries, notes, or ''Eureka!'' reports generated by participants during the course of their data analysis, in which they will record insights and discoveries they make while exploring the dataset. We will also encourage recording of usability issues, the reporting and acknowledgment of which can be used to improve Overview.
In addition to the dataset, participants will be given a rough idea about what the dataset is about (subject matter), its provenance, and its high-level metadata. As in previous work (<<cite Saraiya2004>>, <<cite O'Brien2010>>), we will ask participants to generate a list of questions about the dataset they expect to answer during the course of their analysis prior to the start of their analysis. Following their analysis, we will request that participants reflect on this original list and determine whether they were able to answer their original questions. This can be cross-checked with their notes and diary/journal entries, to verify if their recorded insights match answers to these questions.
We will request and collect screenshots from their analysis, which will include visualizations in Overview, annotated documents, and tables/spreadsheets. Overview will also generate usage logs, which can be used to supplement or explain diary and journal entries.
''Show-and-Tell exit interview'': students will conclude each protocol (with Overview, search only), with an interview. In this interview, the participant guides us through their analysis process. This can be facilitated with Skype Premium screen-sharing. For local participants, this can be done in person with basic screen capture recordings.
Finally, should participation in this study culminate in a course assignment, we will also collect this final assignment, along with instructor comments and the grade it receives.
!!!Data analysis methods
Based on the findings and reflections of <<cite North2006>> and <<cite North2011>>, data analysis methods will emphasize qualitative comparisons between subjects, datasets, and study conditions (with Overview, without Overview). Initial coding of insights can be done in the spirit of [[Grounded Theory]] <<cite Charmaz2006>>. We will not undertake a full-scale grounded-theory analysis as our goal is not to construct a theory of document mining, but to evaluate Overview, comparing it to other approaches without Overview; (constructing a theory of document mining is the goal of [[our other study|Document mining in data journalism]]). Despite this, the constant comparative and initial coding techniques of grounded theory are useful for categorizing qualitative data in such a way that domain expertise on behalf of the coders is not strictly necessary. Once coded, our thematic categories of insights, those related to the dataset, can be verified by faculty-member domain experts. Structural categories, such as breadth vs. depth, or summarization insights, can be reliably coded by us the researchers.
''Open coding vs. theoretical coding?'': An alternative to the initial open coding of insights in the style of [[Grounded Theory]] is that of applying existing ~InfoVis / VA theory to a set of codes for categorizing insights. While <<cite North2011>> uses the former, they suggest the latter in their related work section, namely using <<cite Amar2005>>'s analytical activity framework to code the insights. This may not cover the range of all possible insights, and may force insights into preexisting categories. The <<cite Amar2005>> framework may be too low-level, at the level of interface interaction and visual perception. The same may be true of <<cite Shneiderman1996>>'s taxonomy. Alternatively, other frameworks or taxonomies such as <<cite Amar2004a>>, and <<cite Pirolli2009>> may be too high-level, not accounting for mid-level domain-specific categories of insights emerging from the particular dataset in use.
This last question could very well be the subject of another paper altogether. In the meantime, codes should begin at an open, low-level. Being theoretically agnostic / unaware is not realistic. Theory may earn its way into the analysis (see <<cite Charmaz2006>>, <<cite Furniss2011>>), but it won't be forced or shaped at the initial coding stage.
!!References
<<bibliography>>
Analytic Gaps (<<cite Amar2004a bibliography:Bibliography>>) are barriers to higher-level analytic tasks (such as decision making, learning). There are two types of gaps:
*''Rationale gaps'' between perceiving a relationship and appreciating the utility and confidence of the relationship.
*''Worldview gaps'' between what information is shown to a user and what information is hidden: the difference needed to make a conclusion and decision based on the data.
The iterative, nonlinear, and evolutionary dialogue between human analyst and technology, encompassing the issue being addressed, information relevant to the issue gathered by the analyst, evolving knowledge about the issue, which is dynamic, from heterogeneous sources, incomplete, often deceptive. The success of which is dependent upon both the strengths system and human analyst, who must enact/engage convergent as well as divergent thinking, maintain multiple possible assumptions and biases, support and manipulate abstractions of lower-order [[Reasoning Artifacts]].
Source: <<cite Thomas2005 bibliography:Bibliography>>
An positivist, empirical, qualitative methodology for collecting and analyzing data. Beginning with an initial hypothesis, deviant cases that don't fit the hypothesis are actively sought out for the purposes of inductively building a better and more accurate hypothesis, one that generalizes to more cases. Analysis and reformation of the hypothesis is ongoing throughout data collection.
Must be differentiated from deductive logic, Karl Popper's principle of falsification, and hypothesis testing. Handles uncertainty by expanding/changing the initial hypothesis.
Analytical Induction can be complimentary to quantification of results and inferential statistics, as it strives for generalization and universal depiction of a phenomena or causal pattern.
Should also be distinguished from [[Grounded Theory|Grounded Evaluation]], in which no initial hypothesis/theory is present at the outset of data collection. However Analytical Induction shares the aspect of co-occurring data collection and analysis with GT. Furthermore, [[Grounded Theory|Grounded Evaluation]] does not aspire to generalize beyond the context of the phenomena being studied, as it does not aspire to universal truths. Rather, it is about shared interpretations about the constructions being studied, intersubjectivity, and a high degree of specificity.
!!References:
*Original ref: Znaniecki, F. (1934). The method of sociology. New York: Farrar & Rinehart.
**also: Robinson, W. S. (1951). The logical structure of analytic induction. American Sociological Review, 16, 812818.
*[[Analytical Induction|http://www.youtube.com/watch?v=SizaG3KKAp4]]
*[[Wikipedia|http://en.wikipedia.org/wiki/Analytic_induction]]
*<<cite Ratcliff bibliography:Bibliography>>
<<bibliography>>
An autobiographical genre of writing and research, a diary method involving the assembly of design portfolios, design and process critique. This process helps communicate design decisions and possibilities to collaborators and clients more effectively:
>"[...] a helpful framework for reflecting on design ideas emerging from [a] diverse and conflicting body of knowledge".
Source: <<cite Lloyd2011 bibliography:Bibliography>>
!!Bad Stats are Miscommunicated Stats
Pierre Dragicevic - [[aviz.fr/badstats|http://www.aviz.fr/badstats]]
*The tyranny of the discontinuous mind - Dawkins 2011. Dichotomous thinking.
*NHST criticism - Rosevloom 1960, Kline 2004, Fisher 1956
*Cliff effect Rosenthal and Gaito 1963
*Dance of p values. Geoff Cumming. P-intervals.
*Non-standardized effect sizes
!!Evaluating User Behavior and Strategy During Visual Exploration
Khairi Reda, Andrew Johnson, Jason Leigh, Michael Papka
*Cited Sedig 12.
*Weighted Node Link diagrams of cognitive states and interactive states, comparing small and large screen displays.
!!~Value-Driven Evaluation of Visualizations
John Stasko
*Value = Time + Insights + Essence + Confidence
*Essence of the data = inference? Confidence = probability?
!!Benchmark Data for Evaluating Visualization and Analysis Techniques for Eye Tracking for Video Stimuli
Kuno Kurzhals, Daniel Weiskopf
!!Evaluating Visual Analytics with Eye Tracking
Kuno Kurzhals, Brian Fisher, Daniel Weiskopf, Michael Burch
!!Towards Analyzing Eye Tracking Data for Evaluating Interactive Visualization Systems
Tanja Blascheck, Thomas Ertl
*Towards in title = haven't yet done any work.
*Brian Fisher: Dana Ballard on bridging cognitive activities and eye tracking.
*Margot Pohl: eye tracking and individual differences.
*Human coded eye tracking patterns. ~ChronoViz software for coding eye tracking data.
*Cartography and eye tracking. Andrienkos use of eye tracking for maps.
!!Gamification as a Paradigm for the Evaluation of Visual Analytics Systems
Nafees Ahmed, Klaus Mueller (Stony Brook)
*Disguise game. Play online.
!!Crowdster: Enabling Social Navigation in Web-based Visualization using Crowdsourced Evaluation
Yuet Ling Wong, Niklas Elmqvist
!!Repeated Measures Design in Crowdsourcing-based Experiments for Visualization
Alfie ~Abdul-Rahman, Karl Proctor, Brian Duffy, Min Chen
*BELIV buzzword bingo time! Crowdsourcing, gamification.
!!Misc
*hubertngu@gmail.com - [[the framework (UBC)|http://beliv.cs.univie.ac.at/Framework-UBC.pdf]]. To reduce barriers to psychophysical experiments in visualization. Ron Rensink, Lane Harrison involvement
!References
<<bibliography BELIV-14 showAll>>
{{{
@inproceedings{Guiard2006,
address = {New York, New York, USA},
author = {Guiard, Y. and ~Beaudouin-Lafon, M. and Du, Y. and Appert, C. and Fekete, J. D. and Chapuis, O.},
title = {{Shakespeare's complete works as a benchmark for evaluating multiscale document navigation techniques}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168165},
number = {2},
pages = {1},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168165},
year = {2006}
}
@inproceedings{Melancon,
author = {Melancon, G.},
title = {{Just how dense are dense graphs in the real world? A methodological note}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
keywords = {edge density,els,graph mod-,information visualization,interface evaluation,random generation,real world exam-},
year = {2006}
}
@inproceedings{Whiting2006,
author = {Whiting, M. A. and Cowley, W. and Haack, J. and Love, D. and Tratz, S. and Varley, C. and Wiessner, K.},
title = {{Threat Stream Data Generator: Creating the Known Unknowns for Test and Evaluation of Visual Analytics Tools}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {0--2},
year = {2006}
}
@inproceedings{Shneiderman2006,
author = {Shneiderman, B. and Plaisant, C.},
title = {{Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168158},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168158},
year = {2006}
}
@inproceedings{Bertini2006,
author = {Bertini, E. and Santucci, G.},
title = {{Visual Quality Metrics}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--5},
year = {2006}
}
@inproceedings{Valiati2006,
author = {Valiati, E. R. A. and Pimenta, M. S. and Freitas, C. M. D. S.},
title = {{A taxonomy of tasks for guiding the evaluation of multidimensional visualizations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--6},
publisher = {ACM},
url = {http://portal.acm.org/citation.cfm?id=1168169},
year = {2006}
}
@inproceedings{Ellis2006,
author = {Ellis, G. and Dix, A.},
title = {{An explorative analysis of user evaluation studies in information visualisation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168152},
pages = {1},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168152},
year = {2006}
}
@inproceedings{Rester2006,
author = {Rester, M. and Pohl, M.},
title = {{Methods for the Evaluation of an Interactive ~InfoVis Tool Supporting Exploratory Reasoning Processes}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@inproceedings{Henry2006a,
author = {Henry, N. and Fekete, J. D.},
title = {{Evaluating Visual Table Data Understanding}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@inproceedings{Goodell2006,
author = {Goodell, H. and Chiang, C. H. and Kelleher, C. and Baumann, A. and Grinstein, G.},
title = {{Metrics for Analyzing Rich Session Histories}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {2--6},
year = {2006}
}
@inproceedings{Andrews2006,
author = {Andrews, K.},
title = {{Evaluating information visualisations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168151},
pages = {1--5},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168151},
year = {2006}
}
@inproceedings{Zuk2006,
author = {Zuk, T. and Schlesier, L. and Neumann, P. and Hancock, M. S. and Carpendale, S.},
title = {{Heuristics for information visualization evaluation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168162},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168162},
year = {2006}
}
@inproceedings{Mazza2006,
author = {Mazza, R.},
title = {{Evaluating Information Visualization Applications with Focus Groups : the ~CourseVis experience}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@inproceedings{Ardito2006,
author = {Ardito, C. and Buono, P. and Costabile, M. F. and Lanzilotti, R.},
title = {{Systematic Inspection of Information Visualization Systems}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@inproceedings{Lee2006,
author = {Lee, B. and Plaisant, C. and Parr, C. S. and Fekete, J. D. and Henry, N.},
title = {{Task Taxonomy for Graph Visualization Categories and Subject Descriptors}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
}}}
{{{
@inproceedings{Heera,
author = {Heer, J.},
title = {{Evaluating Visualizations to Unearth Behavior and Insight}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Gerken,
author = {Gerken, J. and Bak, P. and Jetter, H. C. and Klinkhammer, D. and Reiterer, H.},
title = {{How to use interaction logs effectively for usability evaluation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {5--7},
year = {2008}
}
@inproceedings{Faisal2008,
author = {Faisal, S. and Craft, B. and Cairns, P. and Blandford, A.},
title = {{Internalization, Qualitative Methods, and Evaluation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--8},
year = {2008}
}
@inproceedings{Lam2008,
author = {Lam, H. and Munzner, T.},
title = {{Increasing the utility of quantitative empirical studies for meta-analysis}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1377966.1377969},
pages = {1},
url = {http://portal.acm.org/citation.cfm?doid=1377966.1377969},
year = {2008}
}
@inproceedings{Yi2008,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, Julie A},
title = {{Understanding and Characterizing Insights: How Do People Gain Insights Using Information Visualization?}},
booktitle = {Proc. of BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV '08)},
keywords = {evaluation,information visualization,insight,sensemaking},
pages = {1--6},
year = {2008}
}
@inproceedings{Rijsberman,
author = {Rijsberman, M.},
title = {{Proxies for Clinical Effectiveness in Genetic Information Visualization}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Latulipe,
author = {Latulipe, C.},
title = {{Measuring Exploration Coverage and Evaluating Refinding Mechanisms}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {0--1},
year = {2008}
}
@inproceedings{Robertsona,
author = {Robertson, G.},
title = {{~BEyond Time and Errors 㩴ion Statement}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {2--3},
year = {2008}
}
@inproceedings{Isenberg2008,
author = {Isenberg, P. and Zuk, T. and Collins, C. and Carpendale, S.},
title = {{Grounded evaluation of information visualizations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1377966.1377974},
url = {http://portal.acm.org/citation.cfm?doid=1377966.1377974},
year = {2008}
}
@inproceedings{Tory2008,
author = {Tory, M. and ~Staub-French, S.},
title = {{Qualitative Analysis of Visualization: A Building Design Field Study}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Bertini2008,
author = {Bertini, E. and Perer, A. and Plaisant, C. and Santucci, G.},
title = {{BELIV'08: ~BEyond time and errors: novel evaluation methods for information visualization}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {3913--3916},
url = {http://portal.acm.org/citation.cfm?id=1358955},
year = {2008}
}
@inproceedings{Shen2008,
author = {Shen, X. and Moere, A. V. and Eades, P.},
title = {{The ~Long-Term Evaluation of Fisherman in a ~Partial-Attention Environment}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--6},
year = {2008}
}
@inproceedings{Connell,
author = {Connell, T. A. O. and Choong, Y. Y.},
title = {{~User-Centered Evaluation Methodology for Interactive Visualizations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Andrews2008,
author = {Andrews, K.},
title = {{Evaluation Comes in Many Guises}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {7--8},
year = {2008}
}
@inproceedings{Santos,
author = {Santos, B. S.},
title = {{Evaluating Visualization techniques and tools: what are the main issues ?}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Valiati2008,
author = {Valiati, E. R. A. and Freitas, C. M. D. S. and Pimenta, M. S.},
title = {{Using Multi-dimensional In-depth ~Long-term Case Studies for information visualization evaluation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--7},
year = {2008}
}
@inproceedings{Whiting2008,
author = {Whiting, M. A. and Haack, F. and Varley, C.},
title = {{Creating Realistic , ~Scenario-Based Synthetic Data for Test and Evaluation of Information Analytics Software}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
number = {c},
year = {2008}
}
@inproceedings{Henry,
author = {Henry, N. and Elmqvist, N. and Fekete, J. D.},
title = {{A Methodological Note on Setting-up Logging and Replay Mechanisms in ~InfoVis Systems}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Street2008,
author = {Huang, W. and Eades, P. and Hong, S. H.},
title = {{~BEyond Time and Error : A Cognitive Approach to the Evaluation of Graph Drawings}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--8},
year = {2008}
}
@inproceedings{Mcnee2008,
author = {~McNee, S. M. and Arnette, B.},
title = {{Productivity as a Metric for Visual Analytics: Reflections on ~E-Discovery}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--6},
year = {2008}
}
}}}
{{{
@inproceedings{Scholtz2010,
author = {Scholtz, J.},
title = {{Developing Qualitative Metrics for Visual Analytic Environments}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Mayr2010,
author = {Mayr, E. and Smuc, M. and Risku, H.},
title = {{Many roads lead to Rome: Mapping users' problem-solving strategies.}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Saraiya2010,
author = {Saraiya, P. and North, C. and Duca, K.},
title = {{Comparing Benchmark Task and Insight Evaluation Methods on Timeseries Graph Visualizations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Conversy2010,
author = {Conversy, S. and Hurter, C.},
title = {{A Descriptive Model of Visual Scanning}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Riche2010,
author = {Riche, N. H.},
title = {{BEyond system logging: human logging for evaluating information visualization}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--3},
year = {2010}
}
@inproceedings{Kosara2010,
author = {Kosara, R. and Ziemkiewicz, C.},
title = {{Do Mechanical Turks Dream of Square Pie Charts?}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Goldberg2010,
author = {Goldberg, J. H. and Helfman, J. I.},
title = {{Comparing Information Graphics: A Critical Look at Eye Tracking}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Kang2010,
author = {Kang, Y. A. and Gorg, C. and Stasko, J.},
title = {{Pragmatic Challenges in the Evaluation of Interactive Visualization Systems}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Pohl2010,
author = {Pohl, M. and Wiltner, S. and Miksch, S.},
title = {{Exploring Information Visualization - Describing Different Interaction Patterns}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Weaver2010,
author = {Weaver, C.},
title = {{Look Before You Link: Eye Tracking in Multiple Coordinated View Visualization}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Matzen2010,
author = {Matzen, L. and Mcnamara, L. and Cole, K. and Bandlow, A. and Dornburg, C. and Bauer, T.},
title = {{Proposed Working Memory Measures for Evaluating Information Visualization Tools}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--2},
year = {2010}
}
@inproceedings{Smuc2010,
author = {Smuc, M. and Mayr, E. and Risku, H.},
title = {{Is Your User Hunting or Gathering Insights? Identifying Insight Drivers Across Domains}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Jeong,
author = {Jeong, D. H. and Green, T. M.},
title = {{Comparative Evaluation of Two Interface Tools in Performing Visual Analytics Tasks}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Chang2010,
author = {Chang, R. and Ziemkiewicz, C. and Pyzh, R. and Kielman, J. and Ribarsky, W.},
title = {{~Learning-Based Evaluation of Visual Analytic Systems}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Kinnaird,
author = {Kinnaird, P. and Romero, M.},
title = {{Focus Groups for Functional ~InfoVis Prototype Evaluation: A Case Study}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Whiting2009,
author = {Whiting, M. A. and Haack, J. and Varley, C.},
title = {{Generating a Synthetic Video Dataset}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Liiv2010,
author = {Liiv, I.},
title = {{Towards ~Information-Theoretic Visualization Evaluation Measure: A Practical example for Bertin's Matrices}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Yi2010,
author = {Yi, J. S.},
title = {{Implications of Individual Differences on Evaluating Information Visualization Techniques}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Elmqvist2010,
author = {Elmqvist, N.},
title = {{Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in ~InfoVis Evaluation}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {2--3},
year = {2010}
}
@inproceedings{Heer2010,
author = {Heer, J.},
title = {{Visualization Evaluation of the Masses, by the Masses, and for the Masses}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {13--14},
year = {2010}
}
@inproceedings{Wickham2010a,
author = {Wickham, H.},
title = {{How is a graphic like pumpkin pie? A framework for analysis and critique of visualisations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Sedlmair2010,
author = {Sedlmair, M.},
title = {{Evaluating Information Visualization in Large Companies: Challenges, Experiences and Recommendations}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
}}}
{{{
@inproceedings{Slingsby2012,
author = {Slingsby, A. and Dykes, J.},
title = {{Experiences in involving analysts in visualisation design}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Silva2012,
author = {Silva, I. C. S. and Freitas, C. M. D. S. and Sasso, D. and Santucci, G.},
title = {{An integrated approach for evaluating the visualization of intensional and extensional levels of ontologies}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Cushing2012,
author = {Cushing, J. B. and Hayduk, E. and Walley, J. and Winters, K. and Lach, D. and Stafford, S.},
title = {{Which visualizations work, for what purpose, for whom? Evaluating visualizations of terrestrial and aquatic systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Jackson2012,
author = {Jackson, B. and Coffey, D. and Thorson, L. and Schroeder, D. and Ellingson, A. M. and Nuckley, D. J. and Keefe, D. F.},
title = {{Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Anderson2012,
author = {Anderson, E. W.},
title = {{Evaluating scientific visualization using cognitive measures categories and subject descriptors}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Peck2012,
author = {Peck, E. M. and Yuksel, B. F. and Harrison, L. and Ottley, A. and Chang, R.},
title = {{ICD 3: Towards a 3-dimensional model of individual cognitive differences}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Endert2012,
author = {Endert, A. and North, C.},
title = {{Interaction junk: User interaction-based evaluation of visual analytic systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Cottam2012,
author = {Cottam, J. A. and Lumsdaine, A.},
title = {{Spatial autocorrelation-based information visualization evaluation}},
volume = {0},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Dasgupta2012,
author = {Dasgupta, A. and Kosara, R.},
title = {{The importance of tracing data through the visualization pipeline}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Gleicher2012,
author = {Gleicher, M.},
title = {{Stop the evaluation arms race! A call to evaluate visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Meyer2012,
author = {Meyer, M. and Sedlmair, M. and Munzner, T.},
title = {{The four-level nested model revisited: blocks and guidelines}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Elmqvist2012,
author = {Elmqvist, N. and Yi, J. S.},
title = {{Patterns for visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Scholtz2012,
author = {Scholtz, J. and Whiting, M. A. and Plaisant, C. and Grinstein, G.},
title = {{A reflection on seven years of the VAST challenge}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Mcnamara2012,
author = {McNamara, L. A. and Orlando-Gay, N.},
title = {{Reading, sorting, marking, shuffling: Mental model formation through information foraging}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Kaastra2012,
author = {Kaastra, L. T. and Arias-Hernandez, R. and Fisher, B.},
title = {{Evaluating analytic performance}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Kim2012,
author = {Kim, S. H. and Yun, H. and Yi, J. S.},
title = {{How to filter out random clickers in a crowdsourcing-based study?}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Forsell2012,
author = {Forsell, C. and Cooper, M.},
title = {{Questionnaires for evaluation in information visualization}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Pohl2012,
author = {Pohl, M.},
title = {{Methodologies for the analysis of usage patterns in information visualization}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
}}}
!~BELIV-12 Proceedings
*[[17MB zip|https://dl.dropbox.com/u/6397998/beliv-12.zip]] (16 papers, see full reference list below)
!Organizer notes
*[[Day 1 Wrap-up|https://docs.google.com/presentation/d/1RYixe-QSUGqn6o0j1c8IQYzwQ8hSpaxPxbLvnuYP-lg/edit]] (slide deck)
!Session notes
!!Evaluation and Design (~InfoVis): How do we learn from users at the design stage to correct mistakes before building a full prototype?
!!!Presenters:
*Aidan Slingsby (''AS'') (<<cite Slingsby2012 bibliography:BELIV-12 showAll>>)
*Isabel Siqueira Silva (''IS'') (<<cite Silva2012 showAll>>)
*//chair//: Heidi Lam (''HL'')
!!!Notes:
//Evaluation at Design (~InfoVis): How do we learn from users at the design stage to correct mistakes before building a full prototype?// (''AS''):
*demonstrate / show universe of possibilities to users, those close to their domain and those further away, possibilities they haven't thought of; broaden horizons of users; discourage users from being fixated on one technique they happen to like ("I want that one"); culture of mutual contribution;
*pitfalls: fixating on single design
*side-benefits: diverging stage may lead to other future projects
*followed by iterative design stage, generate many disposable design prototypes; agile development; analytical task to endorse or dispose of design alternatives
//An Integrated Approach for Evaluating the Visualization of Intensional and Extensional Levels of Ontologies// (''IS''):
*furthering grounded evaluation (Isenberg BELIV '08) for evaluation of visualizations for ontologies; involvement of domain expert users
!!!Discussion:
*learning from mistakes: designers can involve users early with disposable prototypes and awareness workshops:
*''HL'': visualization tools are complicated, how do you develop prototypes incorporating complicated interaction techniques - requires some time investment; danger of becoming attached / unwilling to throw away
**''AS'': extending existing tools as a starting point; development time varies for prototypes
*''Q'': particular prototyping tools / techniques? lightweight? toolkits? what's different about the prototyping lifecycle in ~InfoVis than in traditional HCI approach (i.e. Buxton's sketching)
**''AS'': Processing (language), using user's data early, first stage is not a complete tool, but single isolated techniques, interaction limited; complements drawing / sketching
*''HL'': re: awareness workshops; shows users D3 example page; may not be useful for user task, but helps to inspire / broaden
**''AS'': emphasis on task - what the user wants to find out, not the end product visualization
*''Q'': expert users understand complex data (e.g. ontologies), they often "get it", what about novices? Do experts understand novice behaviour?
**''AS'': not building complete systems, but small components; experts tend to have more interesting questions, better understanding of tasks;
**''Jason Dykes'': visualization awareness workshops are passive
*''Q'': integrations of disparate components, esp. when feedback conflicts, components don't naturally integrate w/ one another;
**''HL'': how do you isolate part of the data? How are tasks representative enough to scale to larger tasks?
**''AS'': high-level aspirations 稭level tasks; focus on low-level tasks doesn't mean that users lose sight of high-level tasks
*''Enrico Bertini'': what's different/unique about visualization? nothing proposed here is different from 20-30 years of HCI research
*''Michael Sedlmair'': it's about the data - real user data is necessary; high-level tasks are not defined in the minds of users; good design
**''JD'': good design depends on structure of data; users' needs change as they see structure in the data that wasn't visible before; specific initial requirements may not be the ideal set of tasks; prototyping helps users to better define their tasks
!!Evaluation and Design (~SciVis): How do we learn from users at the design stage to correct mistakes before building a full prototype?
!!!Presenters:
*Judy Cushing (''JBC'' - Evergreen college, Oregon State) (<<cite Cushing2012 showAll>>)
*Bret Jackson (''BJ'') (<<cite Jackson2012 showAll>>)
*//chair//: Tobias Isenberg (''TI'')
!!!Notes:
//Which Visualizations Work, for What Purpose, for Which Audiences? Visualization of Terrestrial and Aquatic Systems (VISTAS) Project ಥliminary Report// (''JBC'')
*evaluation incorporating social science methods, working with ecology scientists
*~SciVis: shows something about the outside world (~InfoVis does not?) - is it different from ~InfoVis? allows generation of hypothesis for spatial and temporal scales
*emphasis on collaborative workers
*wait to involve domain scientist until you can offer something 6-9 mo. (too late?)
*scientists use vis tools to "advance their science and offer new insights" - what are insights?
*$850K from NSF to build a prototype, not a production tool; is this more expensive than an ~InfoVis tool?
//Toward Mixed Method Evaluations of Scientific Visualizations and Design Process as an Evaluation Tool// (''BJ'') - Slice WIM (Coffey et al TVCG 2011):
*designed for expert users;
*quantitative user study based on low-level navigation study; didn't evaluate full system, too low-level (time and error)
*higher-level qualitative feedback from expert users; but requires fully-implemented system
*alternative: design critiques - e.g. vector fields Jackson et al 03; design hasn't caught on in broader ~SciVis field relative to quant. user studies;
*spine motion visualization: 320 design sketches, depth and breadth of ideation, each critiqued
*~SciVis, much harder to to sketch ideas in 3D, 4D - needs better ideation tools
*diverging stage: uncertainty, patterns, insights; converging stage: clarity/focus
*ideation tools: design caves, wizard-of-Oz prototyping, sketch-based tools
*call for ideation and critique sections in research papers
!!!Discussion:
*''TI'': ~SciVis evaluation harder than ~InfoVis, other interfaces in broader HCI? Tool complexity. how is evaluation being treated and expected in ~SciVis? A certain type is expected (time and error)
**''JBC'': hiring a social scientist, inductive qualitative evaluation;
*''Q'': we need better tools, provenance tracking of design ideas; what are the features for tracking those tools?
**''BJ'': Illustrator for sketching, being able to show users' real data, digital format; no easy way to search them (folder navigation), needs more metadata; a meta-visualization is needed
*''Michael Sedlmair'': precious time of collaborators - can't show hundreds of design in single session; subdomains are more closely related than we think;
**''BJ'': weekly meetings with collaborators; relying on background as designers, self-critiquing
*''Enrico Bertini'': ~SciVis has more obvious mapping from data to visual dimensions; domain scientists may have more precise expectations of visual representations
**''BJ'': differences b/w scientific domains may be more obvious in ~SciVis than in ~InfoVis
**''JBC'': there are still some degrees of freedom for deciding upon visual abstractions; domain users precise expectations often change, even in ~SciVis designs
*''Q'': evaluation as a first-class contribution - where/how to publish?
*''Laura ~McNamara'': training, ideation addressing visual perception, memory (replacing need for low-level quant. studies?)
**''BJ'': graphic designers are well-trained, their critique is ongoing and incorporates this; advocates mixed-method evaluation, still a need for low-level quant. studies
*''Q'': how to categorize design sketches; how to tease out successful/unsuccessful elements of sketches?
**''BJ'': no formal tracking / categorization of ideas, but an overall understanding of design stages.
*''JD Fekete'': artery visualization (Borkin et al. ~InfoVis 2011); 3D not always better for specific tasks; exploding design space by considering 2D representations; design sketches may be biased early on
!!Evaluating visualizations: How can we measure visualization?
!!!Presenters:
*Robert Kosara (''RK'' - UNCC, Tableau) (<<cite Dasgupta2012 showAll>>)
*Joseph Cottam (''JC'') - CREST / Indiana U (<<cite Cottam2012 showAll>>)
*//chair//: Adam Perer (''AP'')
!!!Notes:
//Importance of --tracing-- (chasing) data through the visualization pipeline// (''RK'')
*how does program know what is a better visualization design?
*data mapping > visual mapping > measure visual representation > visual uncertainty: perception > cognition > uncertainty analysis (feedback loop back into screen-space metrics, measuring of visual uncertainty, data mapping, visual mapping)
*pargnotsics (Proc ~InfoVis 2010), metrics for parallel coordinates; particular to type of visualization
*introspection happens too late in design pipeline;
//Spatial ~Autocorrelation-Based Information Visualization Evaluation// (''JC'')
*not tied to particular visualization types (scagnostics, pargnostics)
*Chen/Janicke (TVCG 2010) - information theory and vis. types of problems that occur throughout the visualization pipeline, entropy based; mutual information; irrespective of data types / plot types; can lead to good as well as bad plots; has implications for overviews and focus/detail plots - what information to include in the overview vs. detail views
*spatial autocorrelation: direct measure of pattern and space; Moran's I measure of spatial autocorrelation - detecting patterns not only from global averages but also spatial neighbours, control over local patterns - but how to define neighbours?
*data set investigated: D3 examples (of good visualizations), control images, classics, looked at luminance only (currently spatial autocorrelation must operate in an ordered space; able to use Moran's I to distinguish visualizations from non-visualizations, space-filling visualizations and sunburst diagrams from node-link diagrams;
*doesn't yet tie back into underlying data - could lead to finding structure that isn't there? false positives/negatives?
!!!Discussion:
*''AP'': the user wasn't mentioned in this session - removing the users from the evaluation loop?
**''JC'': protecting users' time; visualizations are deployed into the wild, spatial autocorrelation feedback loop should be integrated into software; unforeseen user deployments: e.g. scatterplots on small screens
**''RK'': users and tasks clearly related, users are implicitly part of feedback loop
*''Q'': discussing the loop w/ perceptual psychologists is easy, building/integrating this loop in design is hard -
**''RK'': we need to reach out more to perception/cognition researchers
*''JD Fekete'': user is missing; problem w/ autocorrelation - will tell you something that is not true; Wattenberg song-arcs music visualization: pattens and residuals; the latter is often more interesting, but not spatially autocorrelated; assumptions that the measures you are taking are useful/interesting, but this has yet to be shown;
**''JC'': this is preliminary work; local spatial autocorrelation - pointing out local deviations is part of the feedback loop; intent is to present pieces that //may// be interesting;
*''JD Fekete'': input space is important, overplotting cannot be detected by feedback loop
*''Q'': what is end goal? turning this into an AI complete problem? heading to an oracle? relying on a ML approach to tell us what is interesting
*''Q'': research on the metric (descriptive) vs. research on the user (prescriptive)
**''RK'': building more metrics, integrating into larger evaluation methodology
!!Cognition and evaluation (new metrics / measures): How can we measure user cognition?
!!!Presenters:
*Erik Anderson (''EA'' - Utah) (<<cite Anderson2012 showAll>>)
*Alex Endert (''AE'') (<<cite Endert2012 showAll>>)
*Alvitta Ottley (''AO'') (<<cite Peck2012 showAll>>)
*//chair//: Tobias Isenberg (''TI'')
!!!Notes:
//Evaluating Scientific Visualization Using Cognitive Measures// (''EA''):
*cognitive load (CL) and WM linked
*measures of CL: ~NASA-TLX, eye tracking, EEG
*insight: problem solving strategies - sudden realization (insight) vs. gradual accumulation of data - measuring from EEG requires explicit acknowledgment of problem solving strategy
*combining EEG and eye tracking
//The ~ICD3 Model: Individual Cognitive Differences in Three Dimensions// (''AO''):
*individual difs: locus of control measures, ~fNIRS to measure workload;
*can discern novice and expert behaviour (breadth vs. depth, respectively)
*3-dimensional model of ICD: cognitive trait (locus of control), cognitive state, experience/bias (learned behaviour, confirmation bias, expertise)
*how to design evaluations around these dimensions, ignoring one of the 3 dimensions is incomplete
*Interaction Junk (''AE''):
*likening to chartjunk
*broken metaphors with complex visualizations, dimension reduction
*interaction for sensemaking - extraction, synthesis
*semantic interaction, reducing cognitive distance
*interaction junk = distance from metaphor / insight
!!!Discussion:
*''TI'': ultimately we want to measure insight. How with these techniques?
*''EA'': measure moment of insight with EEG sensors (but what is the insight? degree?) can measure kind of insight (gradual / sudden)
*''Q'': quality of the insight, subjective; you can have the wrong insights
**''EA'': can measure when, but not quality - cannot tease out if insight is about task, data, or tool
*''TI'': more complex systems?
**''AO'': individual differences are not created equally, they interact, and often can't be seen with simple tasks, only difficult tasks and holistic larger systems may bring about these differences.
**''AE'': at this stage it is hard to separate learning about the tool and learning about the dataset
*''Q'': drawbacks/tradeoffs to interaction parameters;
**''AE'': when does direct manipulation not apply / doesn't make sense?
*''Enrico Bertini'': endpoint of personalization research? introducing more complex systems? adaptive visualization systems?
**''AO'': implications for mass-deployment of these tools; personalized software
**''EA'': 1948 study forcing people into certain problem-solving strategies (also ''AO'''s poster - "nudging")
*''Brain Fisher'': personal equation (colour blindness, stereo blindness) - you can impact how users perform, either leaving them to perform very well or at chance; personality differences are additional confounds
*''Michael Sedlmair'': how to define "interaction junk" for imprecise, high-level tasks? Like "exploration"?
**''AE'': exploration is a well-defined task.
**''MS'': but it's high-level. something like "finding an outlier" would be more precisely defined.
*''AO'': complementing these approaches with interaction logging
*''BJ'': implications for interaction junk and software learnability?
**''AE'': maybe all these interactive controls aren't necessary
*''Q'': mood, other factors, data transfer rate under high workload
!!Why evaluate?: What are the goals and motivations of evaluations? How should these be conveyed in reporting evaluation?
!!!Presenters:
*Michael Gleicher (''MG'' - ) (<<cite Gleicher2012>>)
*Miriah Meyer (''MM'') (<<cite Meyer2012 showAll>>)
*//chair//: Enrico Bertini (''EB'')
!!!Notes:
//Stop The Evaluation Arms Race! A Call to Evaluate Visualization Evaluation// (''MG''):
*Why evaluate? reflect a change in thinking. Why ask why? Not whether evaluation is necessary, but what we should evaluate, how evaluation can be done well. Good evaluations can guide and persuade. Are guiding and persuading diamteric opposites?
*guide: non-prescriptive -> actionable
*persuade: vacuous assertion -> compelling argument
*how do you (inform/convince) the (audience) to do some (thing); audience, thing = context
*evaluating evaluations: making them good: actionable, persuasive
//The ~Four-Level Nested Model Revisited: Blocks and Guidelines// (''MM''):
*nested model blocks and guidelines: mappings and comparisons
*what are the actionable guidelines for between-level mappings
!!!Discussion:
*''EB'': what community are we addressing?
**''MG'': our community is too self-serving; we need to be more actionable to
*''Tim Lebo'' (RPI): are the blocks formalized? the recipe book for visualization?
*''Q'': what are the users' tasks? their contexts?
*''Q'': we need to know more about the low-level and high-level composite tasks?
**''Tamara Munzner'': we need a better understanding of mid-level tasks;
**''Laura ~McNamara'': auditor audience evaluation vs. stakeholder evaluation; no mid-level tasks, these are constantly evolving; we should be empowering users to determine these abstractions themselves rather than catalogue themselves
**''Sheelagh Carpendale'': danger of guidelines, formulaic writing; studies never give you anything: precision, generalization, realism; we as a community don't look to realism in studies;
**''MG'': evaluations that are persuasive and realistic are hard; both persuasive and actionable is hard; it won't occur through a single study, but through multiple means
**''Heidi Lam'': if we don't aspire to transferability, we'll never be done, we'll keep evaluating tools/techniques in single contexts without transferable actionability to other contexts
*''John Stasko'': most evaluations are neither persuasive nor actionable, but aimed at refinement, improving one's tool/technique; formative design, included to please reviewers
*''TM'': confirm, refine, reject, propose (DSM); persuade / make actionable - what are the terms? what do we do?
**''EB'': is it necessarily bad to be non-prescriptive or make vacuous assertions?
**''MG'': what about the term "inspire" (is this guidance?)
**''HL'': framework and transferable context?
**''TM'': guidelines helping or hurting?
**''SC'': humans are complex adaptive systems and visualizations are linear systems; we're trying to apply linear methods without acknowledging the full complexity; you can't re-assemble complex adaptive systems
**value to non-actionable evaluation, as it may not be actionable //now//, but research contexts change over time
!!Improving existing methods
!!!Presenters:
*Margit Pohl (''MP'') (<<cite Pohl2012 showAll>>)
*~Sung-Hee Kim (''SHK'') (<<cite Kim2012 showAll>>)
*Camilla Forsell (''CF'') (<<cite Forsell2012 showAll>>)
*//chair//: Heidi Lam (''HL'')
!!!Notes:
//Methodologies for the Analysis of Usage Patterns in Information Visualization// (''MP''):
*applying methodologies from other fields; knowing interaction processes
*log file analysis in other areas of HCI (e.g. web navigation), but doesn't capture cognitive processes; which activities to log? how should log files be interpreted? how should the results generalize? signal/noise: how much interaction is "interaction junk"? are all interactions meaningful - how much can we infer?
//How to Filter out Random Clickers in a ~Crowdsourcing-Based Study?// (''SHK''):
*performance-based rewards, filtering out unfaithful users, experimenting with reward schemes; replicating existing studies
*(Does the add anything new? from Willett/Heer/Agrawala CHI 2012? Heer/Bostock CHI 2010? - they cite Heer/Bostock, not Willett)
*100 faithful users costs $30, not $500
//Questionnaires for Evaluation in Information Visualization// (''CF''):
*why use questionnaires: guide other methods, please reviewers;
*no standard practice for questionnaires in ~InfoVis; we use homegrown questionnaires; what is the validity of these questionnaires? how can they be trusted?
*we as a community needs standardized questionnaires; small, low-dimensional; but this is impossible to generalize
*existing questionnaires focus on control (interaction)?
!!!Discussion:
*''HL'': user-centred design - what makes ~InfoVis unique? ~MTurk, log analysis, and questionnaires emanate from other domains; how do we fit these to ~InfoVis? Can we overfit? go too far? adapt something that is not useful?
**''MP'': we still need to try out methods and report them
**''HL'': but can we decide before trying out?
*''TM'': compare Willett/Heer/Agrawala 2012 to SHK's strategies
**''Q'': are questionnaires publishable? ''TM'': yes, but it would be hard to write!
**''Q'': how can questionnaires from other fields be used?
**''Erik Anderson'': unique aspects of visualization, how can you tease apart personal preferences (aesthetics) and objective quality of a visualization?
!!Novel methods
!!!Presenters:
*Laura ~McNamara (''LM'') (<<cite Mcnamara2012 showAll>>)
*Brian Fisher (''BF'') (<<cite Kaastra2012 showAll>>)
*//chair//: Heidi Lam (''HL'')
!!!Notes:
//Reading, Sorting, Marking, Shuffling: Mental Model Formation through Information Foraging// (''LM''):
//Evaluating Analytic Performance// (''BF''):
*//General Specific//: analytics (general) and analysis (analysis)
*analytics (general): methods, problem classes, extensibility, defined domains
*analysis (specific): methods instantiated in tool, application to particular domain, individual differences, domains evolve
*18 intelligence analysts: interviews, observations of analysts using a text analytics tool for LSA, LDA
*comparing the VA tool against card-sorting exercise (incredible amount of variation)
*role of expertise that gets cut out of LSA/LDA algorithms
*role of information scent, information foraging
*impact of working memory, cognitive load, adaptive/custom searches
*explanatory power, larger than the phenomenon, occurring across fields
*Martin Heidegger: ready at hand vs. present at hand? tools under evaluation are more often present to hand (objective analysis of the tool) rather than ready to hand (is the task supported?)
*see BF's topics in cognitive science article //Evaluating Analytic Performance// (''BF'')
*bilateral observations of behaviour, translations between lab and in-the-wild
*VA is transitional, a trading zone
*new methods and methodologies for laboratory / field studies
*approaching new venues outside of visualization
!!!Discussion:
*''AP'': @LM lessons learned?
**''LM'': ecologies of information (Bonnie Nardi); transferability of results
*''JS Yi'': cognitive sciences resources for ~InfoVis practitioners?
**''BF'': cognitive sciences have no fixed goal, no halting point? cognitive sciences is an esperanto, a trading zone, as is VA; you're already cognitive scientists
**''LM'': no disciplinary boundaries in national labs, may be harder in academics
*''HL'': @LM how do you rescue a failed experiment?
*''MS'': how to apply these methods with the time/logistical constraints that we have?
**''BF'': collaboration, division of methodological expertise
*''JBC'': novices vs. experts?
**''LM'': provenance capture can be useful here
!!New evaluation framework: What can we learn from patterns and templates and apply to visualization evaluation?
!!!Presenters:
*Ji Soo Y (''JSY'') (<<cite Elmqvist2012 showAll>>)
*Georges Grinstein (''GG'') (<<cite Scholtz2012 showAll>>)
*//chair//: Enrico Bertini (''EB'')
!!!Notes:
//Patterns for Visualization Evaluation// (''JSY''):
*design patterns
*//"the future is already here, it's just not evenly distributed"// - William Gibson
*categories of ~InfoVis evaluation patterns (killed categories)
*pair analytics (Fisher, ~Arias-Hernandez 2011)
*pilot studies - should these be reported?
*human blackbox
*luck control
*+8 more
*[[http://visevalpatterns.wikia.com]]
//A Reflection on Seven Years of the VAST Challenge// (''GG''):
*using patterns to evaluate VAST challenge entries
*evaluation has changed over the past 6 years
*looking at other metrics aside from accuracy, creative approaches
*example: using a plagiarism tool to study genetic mutations, using crowdsourcing, card sorting
!!!Discussion:
*''EB'': @GG where do you draw the line re: creative challenge entries
**''GG'': eastern vs. western self-reflective evaluation of cognitive processes (e.g. meditation for insight, longitivity, patience - but larger datasets throws a wrench into this, we have to face the deluge of large data
*''Erik Anderson'': @GG 7 interpretations, and they're all correct. How to address this?
**''GG'': reading the same data, interpreting it differently, transferred with beliefs
*''HL'': how to evaluate these evaluation patterns / templates?
**''GG'': uses the pilot study template all the time. We need to write more about these studies, the assumptions held; replicability of pilot studies; pilot studies bias us and in turn affect the resulting study design
**''JSY'': are design patterns evaluated in software engineering? No. but they work. People have found them to be useful. Where should we find new evaluation patterns?
!!Closing remarks
*speaker: Tobias Isenberg (''TI'')
*BELIV '12: continues tomorrow, 9:15-17:00 @Seattle Public Library
*potential discussion topics
*[[BELIV notes|http://beliv.org]]
!!Day 2 discussion
A research agenda to address the issues:
*[[Day 2 Ideas|https://docs.google.com/presentation/d/1xySynhMyzxYTLuK808DumxSzB5gMAKq7UwWLJ23GfMg/edit]] (slide deck)
*[[JS Yi's guidance notes|https://docs.google.com/document/d/1IOIzSDCgFwEA0EmY418Xw0o8o6hDLXB-MOc9us9wn4o/edit]] (notes)
*Promote
*Tasks
**mid-level task taxonomies - task abstractions / mappings b/w
**chunking/segmenting tasks for evaluation / prototyping
*En masse eval - scalable evaluation method(ologies)
*Why evaluate? (reproducibilty / how to reuse)
*Guidance
*Rapid prototyping
*How to talk / collab. w/ other disciplines
*Cognition / individual differences
**Acknowledging a split in opinion from cog. psychologists, who ignore individual differences
*Evaluation focus
*Reporting details: evaluating the evaluations
!!!Morning Discussion: Why evaluate?
Participants: Carla Dal Sasso Freitas (''CDSF''), Brian Fisher (''BF''), Heidi Lam (''HL''), Matt Brehmer (''MB''), Erik Anderson (''EA''), Alisa Bandlow (''AB'', Sandia natl. labs), Paul Rosenthal (''PR'')
*Why evaluate? What is the purpose of eval?
**A/B comparison? Is tool/technique A (better) than tool/technique B?
**To make results available? Transferability / generalize to other contexts
*''BF'': re: the philosophy of science, A/B comparisons are not a science
**we need to avoid cogno-babble, neuroscience jargon; we need a consistent vocabulary; what does improve performance mean?
**we need convince (govt to/funding bodies/industry) to fund research. Are head-to-head comparisons the most convincing? They can give strong confidence
**we have to demonstrate what's at stake, that lives depend on performance in some application areas - e.g. medical diagnosis, air traffic control, intelligence analysis
*''BF'': We live in a world of analytics, not argumentation. We need to be explicit about what claims we are making
*Evaluation can show us how a tool operationalizes insight (but what is insight?)
*Recall G. Grinstein's BELIV '12 talk, the need for meditation and insight, the slow gradual accumulation of insight; is a user's silence confusion or meditation? mind-wandering or slow insight?
*Engineers are dismissive of human-in-the-loop issues; we have to convince them that these issues are worth paying attention to
*Should there be a evaluation grand challenge? nothing in //Illuminating the Path// about such a challenge. What would the milestones be? The criteria?
*''BF'' mentions a recent NSF workshop on science of interaction, a movement to de-emphasize the artifact, an emphasis on the interaction.
*Why evaluate must consider the logistics of evaluation, tradeoffs in terms of cost and ease of application.
**Lightweight eval could be something akin to diagnostics in the medical sciences,e.g. a simple blood test before proceeding to more rigorous eval, if needed.
*A good enough evaluation is useful if you don't have the resources / ability to *talk to users, don't want to waste users' time.
**In industry R&D vs. academics - how lightweight is enough? two contexts at odd with one another
**On statistical rigour: worrying about //p// values depends on the question and form of eval, nevertheless we must consider what N is sufficient? We're tied to legacy of HCI/cog. psych research that we inherited.
*We're a translational science. So how to collaborate with other sciences? How to assimilate? This feedbacks into complexity of evaluation.
**At the same time, cognitive science's complementary (component?) fields are changing - (''BF'': //blankets for beaver pelts//?)
**''BF'': If we're creating a new community, we need to question the assumptions of the old community
**Vision Sciences researchers are not aware that people are using their research. They should be made aware. But speaking about applications with basic vision / psyc researchers is akin to //moneychangers in the temple//.
*On decomposing problems: if you change the vis, you change the task (but doesn't change the goal/intent/high-level task?)
*Why evaluate? we want to understand the users better. e.g. ''HL'': do users look for structured data? Why build a tool for a task that users don't actually do?
*Why evaluate? Reducing uncertainty in the design space, find blind spots in the design
*Why evaluate? Communicate a warrant to stakeholders (but how do you communicate it?)
*Likening vis. eval to market research: how to fit technology into people's lives: demonstrate usefulness / utility. Meanwhile, the CHI literature emphasizes usability, aesthetics, novel interaction. Utility not a first-class citizen
**CHI is dysfunctional, if it's not sexy, if its basic research, it won't get in.
**We need basic research that generalizes, e.g. Shneiderman's mantra, 7sted model - but where to publish this research? This work is theoretical, not evaluated, and well-written/argumented - what is the value of the method?
**We need equivalent of Psyc review (what is our venue? methods journals? VAST? IVS?)
㹣 behaviour research methods and evaluation journal - methods papers and journals - we need a methods journal - how many eval papers
*people evaluate here (~InfoVis) to get published - evaluation is implied? evaluation should always be explicit -
*everyone assumes evaluation is straightforward, papers get in, bad habits propagates;
*BF: psychology literature integrating methods - we need a summary / curated methods evaluation techniques (i.e. WM studies, pros, cons, variations)
*HL: seven scenarios paper
*pattern papers, actionable
*do HCI methods work? DHS's //illuminating the path// realized that HCI task analysis led them into a cul-de-sac; but we don't understand how these map to high-level tasks - mid-level tasks
*GOMS, computational theories of mind don't work because we don't have defined tasks
*talking about it changes the task (think aloud) (playing piano example)
*distributed cognition protocol with paired analysis - but does this change the strategy of the user when the tool analyst leaves? does it bias tasks?
*what is ground truth? coverage of analytic space?
*methods sections must be informative / standardized? e.g. APA style guidelines - a minimum
*VAST challenge needs more write-up, standardizing - implications for teaching and reproducibility
*e.g. reporting context, experimental setup; where should it be?
*how much detail should be reported? how do other fields do it? consistent at high-level, you know how to scan it; at lower levels, be more flexibility
*adding to supplemental material - you can't change published papers, but you can update supplemental materials online
*guidelines for quant. usability studies, utility studies, physiological studies
*Shneiderman's mantra is not a guideline
*topics in cognitive science - Hegarty's science of visualization special issue (cog. sci society) - W. Gray
*TVCG S. Carpendale editor
*IVS relationship
*usability considered harmful (buxton), panel, dourish on evaluation
*cog sci the switzerland of research (BF)
*VAST and education, analytical reasoning - perception action loop, visual thinking at young ages
*we need integration, not just a survey (laundry list) - they can be published
*shared language is needed, difficulty of terms used in different terms - vocabulary - confusion kills collaboration - people leave
!!!Afternoon Discussion: Tasks
Participants: Carla Dal Sasso Freitas (''CDSF''), Matt Brehmer (''MB''), Stefan (~SciVis guy), Alisa Bandlow (''AB'') (Sandia natl. labs)
*are analytical tasks altogether different from other types of tasks?
*should tasks be assembled from low-level ~GOMS-type granularity
*decomposing tasks to plan experiments
*purpose of determining tasks:
**know the domain of the users
**plan experiments / evaluation
**goals/intents/questions and high-level tasks - a high-level goal could be automated, replaced entirely
**determining the level at which users work / think (individual differences)
**some evaluations cannot reliably measure some tasks
**the no right-answer problem, designing synthetic tasks
**careful to not design tasks that aren't representative
**do high-level tasks (search, sensemaking) generalize?
**can you prove/disprove existence of mid-level tasks
**low-level: select
**example of high-level task: diagnosing a patient (but why do you require a diagnosis?)
***next level down: integration,
***next level down: gathering evidence, integrating and comparing within these, discarding intermediate results
**linearity, sequentiality, and specificity increases at lower levels, low variation in duration of tasks, threshold of where you can build throw-away lightweight prototypes, or design lab-style evaluations vs. longitudinal in-situ studies
**high-level tasks: non-linearity,
*study general-purpose tools (Tableau, spotfire, palantir), across domains,specialists with their own data, goals (intents)
**so why are custom domain-specific tools built? for unsupported data types or unsupported tasks? are unsupported tasks low-level or high-level?
*paired analysis
*triangulation of methods to uncover invisible work (~McNamara), no conscious awareness of what the tasks is, verbalized introspection changes the task
*time is not a metric for high-level tasks, e.g. medical diagnosis
*could ongoing measures of cognitive load map to identifiable, individual low-level tasks?
*low-level tasks homogeneous across
**low-level interaction: pointing, steering, selection
**low-level perception: saccades, scanning
!!References
<<bibliography BELIV-12 showAll>>
{{{
@inproceedings{Scholtz2014,
author = {Scholtz, J. and Love, O. and Whiting, M. and Hodges, D. and Emanuel, L. and Stanton, D.},
title = {{Utility evaluation of models}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kaastra2014,
author = {Kaastra, L. T. and Fisher, B.},
title = {{Field experiment methodology for pair analytics}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Brehmer2014a,
author = {Brehmer, M. and Sedlmair, M. and Ingram, S. and Munzner, T.},
title = {{Visualizing dimensionally-reduced data: Interviews with analysts and a characterization of task sequences}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Brehmer2014b,
author = {Brehmer, M. and Carpendale, S. and Lee, B. and Tory, M.},
title = {{Pre-design empiricism for information visualization: Scenarios, methods, and challenges}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kim2014,
author = {Kim, S.-H. and Yi, J. S. and Elmqvist, N.},
title = {{Oopsy-daisy: Failure stories in quantitative evaluation studies for visualizations}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Aupetit2014,
author = {Aupetit, M.},
title = {{Sanity check for class-coloring-based evaluation of dimension reduction techniques}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Blaha2014,
author = {Blaha, L. M. and Arendt, D. L. and Mohd-Zaid, F.},
title = {{More bang for Your research buck: Toward recommender systems for visual analytics}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kriglstein2014,
author = {Kriglstein, S. and Suchy, N. and G\"{a}rtner, J. and Gschwandtner, T.},
title = {{Experiences and challenges with evaluation methods in practice: A case study}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Tarrell2014,
author = {Tarrell, A. and Fruhling, A. and Borgo, R.},
title = {{Toward visualization-specific heuristic evaluation}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Mercun2014,
author = {Mercun, T.},
title = {{Evaluation of information visualization techniques: Analysing user experience with reaction cards}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Abdul-Rahman2014,
author = {Abdul-Rahman, A. and Proctor, K. J. and Duffy, B. and Chen, M.},
number = {i},
title = {{Repeated measures design in crowdsourcing-based experiments for visualization}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Wong2014,
author = {Wong, Y. L. and Elmqvist, N.},
title = {{Crowdster: Enabling social navigation in web-based visualization using crowdsourced evaluation categories and subject descriptors}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Ahmed2014,
author = {Ahmed, N. and Brook, S. and Mueller, K.},
title = {{Gamification as a paradigm for the evaluation of visual analytics systems}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Blascheck2014,
author = {Blascheck, T. and Ertl, T.},
title = {{Towards analyzing eye tracking data for evaluating interactive visualization systems}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kurzhals2014,
author = {Kurzhals, K. and Fisher, B. and Burch, M. and Weiskopf, D.},
title = {{Evaluating visual analytics with eye tracking}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kurzhals2014a,
author = {Kurzhals, K. and Bopp, C. F. and B\"{a}ssler, J. and Ebinger, F. and Weiskopf, D.},
title = {{Benchmark data for evaluating visualization and analysis techniques for eye tracking for video stimuli}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Stasko2014,
author = {Stasko, J.},
title = {{Value-driven evaluation of visualizations}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Reda2014,
author = {Reda, K. and Johnson, A. E. and Leigh, J. and Papka, M. E.},
pages = {1--5},
title = {{Evaluating user behavior and strategy during visual exploration}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Smuc,
author = {Smuc, M.},
title = {{Just the other side of the coin? From error- to insight-analysis}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Ragan2014,
author = {Ragan, E. D. and Goodall, J. R.},
title = {{Evaluation methodology for comparing memory and communication of analytic processes in visual analytics}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Correll2014a,
author = {Correll, M. and Alexander, E. and Albers, D. and Gleicher, M.},
pages = {1--4},
title = {{Navigating reductionism and holism in evaluation}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Winters,
author = {Winters, K. M. and Lach, D. and Cushing, J. B.},
title = {{Considerations for characterizing domain problems}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Rind2014a,
author = {Rind, A. and Aigner, W. and Wagner, M. and Miksch, S. and Lammarsch, T.},
title = {{User tasks for evaluation: Untangling the terminology throughout visualization design and development}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
}}}
<<list filter [tag[bibtex]]>>
/***
|''Name:''|BibTeXPlugin|
|''Description:''|Very incomplete BibTeX implementation to work with bibliographic references|
|''Author:''|Paulo Soares|
|''Version:''|1.5|
|''Date:''|2010-11-11|
|''Source:''|http://www.math.ist.utl.pt/~psoares/addons.html|
|''Overrides''|Story.prototype.refreshTiddler|
|''Documentation:''|[[BibTeXPlugin Documentation|BibTeXPluginDoc]]|
|''License:''|[[Creative Commons Attribution-Share Alike 3.0 License|http://creativecommons.org/licenses/by-sa/3.0/]]|
|''~CoreVersion:''|2.5.0|
***/
//{{{
if(!version.extensions.BibTeXPlugin) { //# ensure that the plugin is only installed once
version.extensions.BibTeXPlugin = {installed: true};
(function($) {
config.macros.cite = {
noReference: "(??)",
refreshTiddler: Story.prototype.refreshTiddler
};
config.macros.cite.handler = function(place,macroName,params,wikifier,paramString,tiddler) {
var pos, cmb = config.macros.bibliography;
if(params.length==0) return;
var entry = params[0];
var args = paramString.parseParams(null,null,false);
var title = getParam(args,"bibliography",null);
if(title) {
this.biblioTiddler = title;
} else {title = this.biblioTiddler;}
title = getParam(args,"thisBibliography",title);
var format = getParam(args,"format",null);
if(format) {
this.format = format;
} else {format = this.format;}
format = getParam(args,"thisFormat",format);
var argsArray = paramString.readMacroParams();
var showAll = ($.inArray('showAll',argsArray) > -1);
if(title && store.tiddlerExists(title)) var bib = cmb.extractEntry(title, entry);
if(bib.content) {
var entries = this.entries;
if($.inArray(entry, entries)==-1) this.entries.push(entry);
entries = this.entries;
pos = $.inArray(entry, entries)+1;
var author = cmb.processNames(bib.content.extract("author"), showAll);
var year = bib.content.extract("year");
var citation = format.replace("author", author);
citation = citation.replace("year", year);
citation = citation.replace("number", pos);
wikify(citation, place);
} else {
wikify(this.noReference, place);
}
}
Story.prototype.refreshTiddler = function(title,template,force){
config.macros.cite.biblioTiddler = null;
config.macros.cite.format = "author (year)";
config.macros.cite.entries = [];
var tiddler = config.macros.cite.refreshTiddler.apply(this,arguments);
return tiddler;
}
config.macros.bibliography = {
article: {fields: ["author", "year", "title", "journal", "volume", "pages"], format: "author (year). title. //journal// ''volume'', pages."},
book: {fields: ["author", "year", "title", "publisher"], format: "author (year). //title//. publisher."},
inproceedings: {fields: ["author", "year", "title", "editor", "booktitle", "pages", "publisher"], format: "author (year). title. In editor //booktitle//, pages. publisher."},
incollection: {fields: ["author", "year", "title", "editor", "booktitle", "pages", "publisher"], format: "author (year). title. In editor //booktitle//, pages. publisher."},
techreport: {fields: ["author", "year", "title", "institution"], format: "author (year). title. Technical report, institution."},
manual: {fields: ["author", "year", "title", "organization"], format: "author (year). //title//. organization."},
unpublished: {fields: ["author", "year", "title"], format: "author (year). //title//. Unpublished."}
};
config.macros.bibliography.handler = function(place,macroName,params,wikifier,paramString,tiddler) {
var cmc = config.macros.cite;
var title = (cmc.biblioTiddler) ? cmc.biblioTiddler : params[0];
if(!title || !store.tiddlerExists(title)) return;
var argsArray = paramString.readMacroParams();
var i, entryText;
var entries = [];
if($.inArray('showAll',argsArray) > -1) {
entryText = this.extractAllEntries(title);
for(i=0; i<entryText.length; i++) {
entries[entries.length] = this.processEntry(entryText[i], i);
}
} else {
for(i=0; i<cmc.entries.length; i++){
entryText = this.extractEntry(title, cmc.entries[i]);
if(entryText) {
entries[entries.length] = this.processEntry(entryText, i);
}
}
}
entries.sort();
wikify(entries[0] , place);
for (i=1; i < entries.length; i++) {
wikify("\n\n" + entries[i] , place);
}
return true;
}
config.macros.bibliography.processNames = function(names, showAll) {
var i, authors = names.split(" and ");
var entry = authors[0];
var numAuthors = authors.length;
var fullEntry = entry;
if (numAuthors==2) {
entry += " and " + authors[1];
fullEntry = entry;
}
if (numAuthors>2) {
fullEntry = entry;
for (i=1; i < numAuthors; i++) {
if (i==numAuthors-1) {fullEntry += " and "} else {fullEntry += ", "};
fullEntry += authors[i];
}
if(showAll) {entry = fullEntry;} else {entry += " et al.";}
}
return entry;
}
config.macros.bibliography.processEntry = function(entry, pos) {
var field, text=entry.content;
var fields={};
fields.number = pos+1;
var type = this[entry.type];
var output = type.format;
for(var i=0; i<type.fields.length; i++){
field = type.fields[i];
switch(field){
case "author":
fields.author = this.processNames(text.extract("author"), true);
break;
case "title":
var url = text.extract("url");
fields.title = text.extract("title");
fields.title = (url=='') ? fields.title : "[[" + fields.title + "|" + url + "]]";
break;
case "editor":
var editor = text.extract("editor");
fields.editor = (editor=='') ? editor : this.processNames(editor,true) + " (Eds.), ";
break;
default:
fields[field] = text.extract(field);
}
output = output.replace(field, fields[field]);
}
return output;
}
config.macros.bibliography.extractEntry = function(title,entry) {
var bib = {type: null, content: null};
var text = store.getTiddlerText(title);
var re = new RegExp('\\s*@(\\w+?)\\{\\s*' + entry + '\\s*,\\s*(.[^@]+)\\}','mi');
var field = text.match(re);
if(field) {
bib.type = field[1].toLowerCase();
bib.content = field[2];
}
return bib;
}
config.macros.bibliography.extractAllEntries = function(title) {
var bib, field, entries = [];
var text = store.getTiddlerText(title);
var bibs = text.match(/\s*@(\w+?)\{\s*(.[^@]+)\}/mgi);
for(var i=0; i<bibs.length; i++){
field=bibs[i].match(/\s*@(\w+?)\{\s*(.[^@]+)\}/mi);
bib = {type: null, content: null};
if(field) {
bib.type = field[1].toLowerCase();
bib.content = field[2];
if(bib.type!='string' && bib.type!='preamble' && bib.type!='comment') entries.push(bib);
}
}
return entries;
}
config.macros.bibliography.extractField = function(field) {
var text = "";
var re = new RegExp('\\s*'+field+'\\s*=\\s*[\\{|"]\\s*(.+?)\\s*[\\}|"]','mi');
var fieldText = this.match(re);
if(fieldText){
text = fieldText[1].replace(/\{|\}/g,'');
if(field!='url') text = text.replace(/-+/g,"-");
}
return text;
}
String.prototype.extract = config.macros.bibliography.extractField;
config.shadowTiddlers.BibTeXPluginDoc="The documentation is available [[here.|http://www.math.ist.utl.pt/~psoares/addons.html#BibTeXPluginDoc]]";
})(jQuery)
}
//}}}
!Description
Very incomplete (and personal) ~BibTeX implementation to work with bibliographic references.
!Usage
First of all, you need to dump your ~BibTeX entries in a tiddler, lets say [[Bibliography]]. The first macro that this plugin provides is {{{cite}}}. As the name suggests it is used to cite a reference such as {{{<<cite Nea:03>>}}} where {{{Nea:03}}} is a ~BibTeX key for a reference. By default it produces this: <<cite Nea:03 bibliography:Bibliography>>.
However, as this was the first citation in this tiddler you must include the name of the tiddler that contains the ~BibTeX entries like this {{{<<cite Nea:03 bibliography:Bibliography>>}}}.
The {{{cite}}} macro produces a full reference if there are at most two authors, otherwise it is abbreviated like this: <<cite Coda:97>>. If you want to force a full reference then you can use the {{{showAll}}} parameter as in {{{<<cite Coda:97 showAll>>}}} which produces this: <<cite Coda:97 showAll>>.
You can also change the way references are displayed with the {{{format}}} parameter. A string can be passed to this parameter where {{{number}}} represents the order of the citation {{{author}}} stands for the authors and {{{year}}} represents the year of publication. So, for example, including {{{format:"([number] author, year)"}}} in a citation would produce citations formatted as <<cite Nea:03 format:"([number] author, year)">>. The format string can even include Tiddlywiki notation such as {{{format:"@@author@@, ''year''"}}} leading to <<cite Nea:03 format:"@@author@@, ''year''">>.
The {{{bibliography}}} and {{{format}}} parameters affect the reference in which they are used and all the following ones. If you need to restrict the scope to a single reference you can use the {{{thisBibliography}}} and {{{thisFormat}}} variants.
The production of a list of cited references is done with the {{{bibliography}}} macro. If you want to make a list of all your ~BibTeX entries then use {{{<<bibliography Bibliography showAll>>}}}.
<<bibliography>>
{{{
@inproceedings{Aupetit2014,
author = {Aupetit, M.},
title = {{Sanity check for class-coloring-based evaluation of dimension reduction techniques}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@book{Kaptelinin2012,
author = {Kaptelinin, V. and Nardi, B.},
title = {{Activity Activity Theory Theory in HCI}},
publisher = {Morgan \& Claypool},
year = {2012}
}
@inproceedings{Scholtz2014,
author = {Scholtz, J. and Love, O. and Whiting, M. and Hodges, D. and Emanuel, L. and Stanton, D.},
title = {{Utility evaluation of models}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kaastra2014,
author = {Kaastra, L. T. and Fisher, B.},
title = {{Field experiment methodology for pair analytics}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Brehmer2014a,
author = {Brehmer, M. and Sedlmair, M. and Ingram, S. and Munzner, T.},
title = {{Visualizing dimensionally-reduced data: Interviews with analysts and a characterization of task sequences}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Brehmer2014b,
author = {Brehmer, M. and Carpendale, S. and Lee, B. and Tory, M.},
title = {{Pre-design empiricism for information visualization: Scenarios, methods, and challenges}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Kim2014,
author = {Kim, S.-H. and Yi, J. S. and Elmqvist, N.},
title = {{Oopsy-daisy: Failure stories in quantitative evaluation studies for visualizations}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Correll2014,
author = {Correll, M. and Alexander, E. and Albers, D. and Gleicher, M.},
pages = {1--4},
title = {{Navigating reductionism and holism in evaluation}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Winters2014,
author = {Winters, K. M. and Lach, D. and Cushing, J. B.},
title = {{Considerations for characterizing domain problems}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@inproceedings{Rind2014,
author = {Rind, A. and Aigner, W. and Wagner, M. and Miksch, S. and Lammarsch, T.},
title = {{User tasks for evaluation: Untangling the terminology throughout visualization design and development}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@article{Boy2014,
author = {Boy, J. and Rensink, R. A. and Bertini, E. and Fekete, J. D.},
title = {{A principled way of assessing visualization literacy}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Dawson2014,
author = {Dawson, J. Q. and Munzner, T. and McGrenere, J.},
title = {{A search-set model of path tracing in graphs}},
journal = {Information Visualization},
year = {2014}
}
@inproceedings{Schmidt2013a,
author = {Schmidt, C. and M쬬er, J. and Bailly, G.},
title = {{Screenfinity: Extending the perception area of content on very large public displays}},
booktitle = {Proc. ACM CHI},
pages = {1719--1728},
year = {2013}
}
@article{Satyanarayan2014a,
author = {Satyanarayan, A. and Heer, J.},
title = {{Lyra: An interactive visualization design environment}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
volume = {33},
year = {2014}
}
@inproceedings{Alexander2014,
author = {Alexander, E. and Kohlmann, J. and Valenza, R. and Witmore, M. and Gleicher, M.},
title = {{Serendip: Topic model-driven visual exploration of text corpora}},
booktitle = {To appear in IEEE VAST},
year = {2014}
}
@article{Cui2014,
author = {Cui, W. and Liu, S. and Wu, Z. and Wei, H.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{How hierarchical topics evolve in large text corpora}},
year = {2014}
}
@article{Sacha2014,
author = {Sacha, D. and Stoffel, A. and Stoffel, F. and Kwon, B. and Ellis, G. and Keim, D.},
journal = {To appear in IEEE TVCG (Proc. VAST)},
title = {{Knowledge generation model for visual analytics}},
year = {2014}
}
@article{Mckenna2014,
author = {McKenna, S. and Mazur, D. and Agutter, J. and Meyer, M.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Design activity framework for visualization design}},
year = {2014}
}
@article{Sedlmair2014,
author = {Sedlmair, M. and Heinzl, C. and Bruckner, S. and Piringer, H. and M\"{o}ller, T.},
title = {{Visual parameter space analysis: A conceptual framework}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Huron2014a,
author = {Huron, S. and Jansen, Y. and Carpendale, S.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Constructing visual representations: Investigating the use of tangible tokens}},
year = {2014}
}
@techreport{Few2014,
author = {Few, S.},
title = {{Distribution displays, conventional and potential}},
year = {2014}
}
@article{Kindlmann2014,
author = {Kindlmann, G. and Scheidegger, C.},
title = {{An algebraic process for visualization design}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@techreport{Wickham2011,
title = {40 years of boxplots},
author = {Wickham, H. and Stryjewski, L.},
note = {http://vita.had.co.nz/papers/boxplots.pdf
year = {2011}
}
@techreport{Few2008a,
author = {Few, S.},
title = {{Dual-scaled axes in graphs: Are they ever the best solution?}},
note = {http://www.perceptualedge.com/articles/visual_business_intelligence/dual-scaled_axes.pdf},
year = {2008}
}
@article{Aigner2011b,
author = {Aigner, W. and Kainz, C. and Ma, R. and Miksch, S.},
title = {{Bertin was right: An empirical evaluation of indexing to compare multivariate time-series data using line plots}},
journal = {Computer Graphics Forum},
number = {1},
pages = {215--228},
volume = {30},
year = {2011}
}
@article{Isenberg2011a,
author = {Isenberg, P. and Bezerianos, A. and Dragicevic, P. and Fekete, J. D.},
journal = {IEEE TVCG (Proc. InfoVis)},
number = {12},
pages = {2469--2478},
title = {{A study on dual-scale data charts}},
volume = {17},
year = {2011}
}
@inproceedings{SantosAmorim2012,
author = {{Santos Amorim}, E. P. dos and Brazil, E. V. and Daniels, J. and Joia, P. and Nonato, L. G. and Sousa, M. C.},
title = {{iLAMP: Exploring high-dimensional spacing through backward multidimensional projection}},
booktitle = {Proc. IEEE VAST},
pages = {53--62},
year = {2012}
}
@article{Saket2014a,
title = {{Node, node-link, and node-link-group diagrams: An evaluation}},
author = {Saket, B. and Simonetto, P. and Kobourov, S. and Borner, K.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Saket2014,
author = {Saket, B. and Simonetto, P. and Kobourov, S.},
title = {{Group-level graph visualization taxonomy}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
volume = {33},
number = {3},
year = {2014}
}
@article{Mirel2014,
author = {Mirel, B. and G沧, C.},
journal = {BMC Bioinformatics},
number = {1},
pages = {117--128},
title = {{Scientists' sense making when hypothesizing about disease mechanisms from expression data and their needs for visualization support}},
volume = {15},
year = {2014}
}
@article{Gorg2013c,
author = {G沧, C. and Liu, Z. and Stasko, J.},
doi = {10.1177/1473871613495674},
journal = {Information Visualization},
pages = {1--11},
title = {{Reflections on the evolution of the Jigsaw visual analytics system}},
volume = {(In press)},
year = {2013}
}
@article{Suchman1995,
author = {Suchman, L.},
title = {{Making work visible}},
journal = {Communications of the ACM},
number = {9},
pages = {56--64},
volume = {38},
year = {1995}
}
@inproceedings{Smith2014,
author = {Smith, A. and Hawes, T. and Myers, M.},
title = {{Hi顲chie: Interactive visualization for hierarchical topic models}},
booktitle = {Proc. ACL Workshop on Interactive Language Learning, Visualization, and Interfaces},
year = {2014}
}
@inproceedings{Albers2014,
author = {Albers, D. and Correll, M. and Gleicher, M.},
title = {{Task-driven evaluation of aggregation in time series visualization}},
booktitle = {To Appear in Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2014}
}
@article{VandenElzen2013,
author = {van den Elzen, S. and van Wijk, J. J.},
doi = {10.1111/cgf.12106},
title = {{Small multiples, large singles: A new approach for visual data exploration}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
month = jun,
number = {3pt2},
pages = {191--200},
url = {http://doi.wiley.com/10.1111/cgf.12106},
volume = {32},
year = {2013}
}
@article{Gregor2013,
author = {Gregor, S. and Hevner, A. R.},
title = {{Positioning and presenting design science research for maximum impact}},
journal = {MIS Quarterly},
number = {2},
volume = {37},
year = {2013}
}
@article{Hevner2004,
author = {Hevner, A. R. and Ram, S. and March, S. T.},
title = {{Design science in information systems research}},
journal = {MIS Quarterly},
number = {1},
pages = {75--105},
volume = {28},
year = {2004}
}
@inproceedings{Chaney2012,
title = {Visualizing topic models},
author = {Chaney, A. J.-B. and Blei, D. M.},
booktitle = {Proc. Intl. AAAI Conf. Weblogs and Social Media (ICWSM)},
pages = {419--422},
year = {2012}
}
@article{Cao2010,
author={Cao, N. and Sun, J. and Lin, L-R. and Gotz, D. and Liu, S. and Qu, H.},
title = {FacetAtlas: Multifaceted visualization for rich text corpora},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
year = {2010},
volume = {16},
number = {6},
pages = {1172--1181}
}
@article{Meyer2014,
author = {Meyer, M. and Sedlmair, M. and Quinan, P. S. and Munzner, T.},
title = {{The nested blocks and guidelines model}},
journal = {Information Visualization},
note = {http://goo.gl/3tGLCw},
volume = {In press}
}
@book{Fry2008,
title={Visualizing Data},
author={Fry, B.},
publisher={O'Reilly},
year={2008}
}
@inproceedings{Kandel2011a,
title = {Wrangler: Interactive visual specification of data transformation scripts},
author = {Kandel, S. and Paepcke, A. and Hellerstein, J. and Heer, J.},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {3363--3372},
year = {2011}
}
@article{Gotz2009,
author = {Gotz, D. and Zhou, M. X.},
title = {{Characterizing users橳ual analytic activity for insight provenance}},
journal = {Information Visualization},
number = {1},
pages = {42--55},
volume = {8},
year = {2009}
}
@article{Wang2007,
author = {Wang, W. and Lu, A.},
title = {{Interactive wormhole detection and evaluation}},
journal = {Information Visualization},
number = {1},
pages = {3--17},
volume = {6},
year = {2007}
}
@article{Kincaid2005,
author = {Kincaid, R. and Ben-Dor, A. and Yakhini, Z.},
title = {{Exploratory visualization of array-based comparative genomic hybridization}},
journal = {Information Visualization},
number = {3},
pages = {176--190},
volume = {4},
year = {2005}
}
@article{McKeon2009,
author = {McKeon, M.},
title = {{Harnessing the web information ecosystem with wiki-based visualization dashboards}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1081--1088},
volume = {15},
year = {2009}
}
@inproceedings{Robinson2008,
author = {Robinson, A. C.},
title = {{Collaborative synthesis of visual analytic results}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {67--74},
year = {2008}
}
@article{Booshehrian2012,
author = {Booshehrian, M. and M\"{o}ller, T. and Peterman, R. M. and Munzner, T.},
doi = {10.1111/j.1467-8659.2012.03116.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {1235--1244},
title = {{Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity In fisheries management decision making}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03116.x},
volume = {31},
year = {2012}
}
@article{Gratzl2013,
author = {Gratzl, S. and Lex, A. and Gehlenborg, N. and Pfister, H. and Streit, M.},
number = {12},
pages = {2277--2286},
title = {{LineUp: Visual analysis of multi-attribute rankings}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Goodwin2013a,
author = {Goodwin, S. and Dykes, J. and Jones, S. and Dillingham, I. and Dove, G. and Duffy, A. and Kachkaev, A. and Slingsby, A. and Wood, J.},
number = {12},
pages = {2516--2525},
title = {{Creative user-centered visualization design for energy analysts and modelers}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Kang2012,
author = {Kang, Y. A. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2869--2878},
title = {{Examining the use of a visual analytics system for sensemaking tasks: Case studies with domain experts}},
volume = {18},
year = {2012}
}
@article{Brehmer2013,
author = {Brehmer, M. and Munzner, T.},
number = {12},
pages = {2376--2385},
title = {{A multi-level typology of abstract visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Gorg2013,
author = {G笠C. and Liu, Z. and Kihm, J. and Choo, J. and Park, H. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {10},
pages = {1646--1663},
title = {{Combining computational analyses and interactive visualization for document exploration and sensemaking in Jigsaw.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/23267206},
volume = {19},
year = {2013}
}
@article{Dou2013a,
author = {Dou, W. and Yu, L. and Wang, X. and Ma, Z. and Ribarsky, W.},
number = {12},
pages = {2002--2011},
title = {{HierarchicalTopics: Visually exploring large text collections using topic hierarchies}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Tilley2008,
author = {Tilley, S. A.},
doi = {10.1007/s10805-008-9058-8},
journal = {Academic Ethics},
number = {2},
pages = {91--104},
title = {{A troubled dance: Doing the work of research ethics review}},
url = {http://link.springer.com/10.1007/s10805-008-9058-8},
volume = {6},
year = {2008}
}
@book{Vicente1999,
author = {Vicente, K. J.},
publisher = {CRC Press},
title = {{Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work}},
year = {1999}
}
@article{Smith2002,
author = {Smith, K. B.},
title = {{Typologies, taxonomies, and the benefits of policy classification}},
doi = {10.1111/j.1541-0072.2002.tb02153.x},
journal = {Policy Studies Journal},
number = {3},
pages = {379--395},
url = {http://doi.wiley.com/10.1111/j.1541-0072.2002.tb02153.x},
volume = {30},
year = {2002}
}
@book{Bailey1994,
author = {Bailey, K. D.},
publisher = {Sage Publications},
title = {{Typologies and Taxonomies: An Introduction to Classification}},
year = {1994}
}
@book{Ware2013,
address = {San Fransisco, CA, USA},
author = {Ware, C.},
edition = {3rd},
pages = {228--237},
publisher = {Morgan Kaufmann},
title = {{Information Visualization: Perception for Design}},
year = {2013}
}
@book{Andrienko2006,
author = {Andrienko, G. and Andrienko, N.},
pages = {715},
publisher = {Springer-Verlag},
title = {{Exploratory Analysis of Spatial and Temporal Data: A Systematic Approach}},
year = {2006}
}
@inproceedings{Lammarsch2012,
author = {Lammarsch, T. and Rind, A. and Aigner, W. and Miksch, S.},
title = {{Developing an extended task framework for exploratory data analysis along the structure of time}},
booktitle = {Proc. Eurographics International Workshop on Visual Analytic (EuroVA)},
pages = {1--5},
url = {http://publik.tuwien.ac.at/files/PubDat\_207621.pdf},
year = {2012}
}
@book{Raskin2000,
author = {Raskin, J.},
title = {{The Humane Interface: New Directions for Designing Interactive Systems}},
publisher = {Addison-Wesley Longman},
year = {2000}
}
@article{Liu2013,
author = {Liu, Y. and Barlowe, S. and Feng, Y. and Yang, J. and Jiang, M.},
doi = {10.1177/1473871612459995},
title = {{Evaluating exploratory visualization systems: A user study on how clustering-based visualization systems support information seeking from large document collections}},
journal = {Information Visualization},
number = {1},
pages = {25--43},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871612459995},
volume = {12},
year = {2013}
}
@inproceedings{Dork2011,
author = {D沫M. and Carpendale, S. and Williamson, C.},
title = {{The information flaneur: A fresh look at information seeking}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1978942.1979124},
pages = {1215--1224},
url = {http://dl.acm.org/citation.cfm?doid=1978942.1979124},
year = {2011}
}
@article{Dork2012,
author = {D沫M. and Riche, N. H. and Ramos, G. and Dumais, S.},
doi = {10.1109/TVCG.2012.252},
title = {{PivotPaths: Strolling through faceted information spaces}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2709--2718},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327277},
volume = {18},
year = {2012}
}
@book{Aigner2011a,
author = {Aigner, W. and Miksch, S. and Schumann, H. and Tominski, C.},
title = {{Visualization of Time-Oriented Data}},
doi = {10.1007/978-0-85729-079-3},
pages = {286},
publisher = {Springer},
url = {http://www.timeviz.net/},
year = {2011}
}
@inproceedings{Erickson2013,
author = {Erickson, T. and Li, M. and Kim, Y. and Deshpande, A. and Sahu, S. and Chao, T. and Sukaviriya, P. and Naphade, M.},
title = {{The Dubuque electricity portal: Evaluation of a city-scale residential electricity consumption feedback system}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466155},
pages = {1203--1212},
year = {2013}
}
@article{Churchill2013,
author = {Churchill, E. F. and Bowser, A. and Preece, J.},
title = {Teaching and learning human-computer interaction: past, present, and future},
journal = {interactions},
volume = {20},
number = {2},
year = {2013},
pages = {44--53},
url = {http://doi.acm.org/10.1145/2427076.2427086},
doi = {10.1145/2427076.2427086},
}
@inproceedings{Lin2013,
author = {Lin, S. and Hanrahan, P.},
title = {{Modeling how people extract color themes from images}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013}
}
@article{Ulwick2002,
author = {Ulwick, A. W.},
journal = {Harvard Business Review},
number = {1},
pages = {91--97, 126},
title = {{Turn customer input into innovation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/12964470},
volume = {80},
year = {2002}
}
@inproceedings{Dunne2013,
author = {Dunne, C. and Shneiderman, B.},
title = {{Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013}
}
@incollection{Holtzblatt1993,
author = {Holtzblatt, K. and Jones, S.},
title = {{Contextual inquiry: A participatory technique for system design}},
booktitle = {Participatory Design: Principles and Practices},
chapter = {9},
editor = {Schuler, D. and Namioka, A.},
pages = {177--210},
publisher = {Lawrence Erlbaum Associates},
year = {1993}
}
@incollection{Norman2007,
author = {Norman, D. A.},
title = {{Filling much needed holes}},
booktitle = {Interactions},
url = {http://jnd.org},
year = {2007}
}
@incollection{Norman2006,
author = {Norman, D. A.},
title = {{Technology first, needs last}},
booktitle = {Technology},
url = {http://jnd.org},
year = {2006}
}
@inproceedings{Grossman2009a,
author = {Grossman, T. and Fitzmaurice, G. and Attar, R.},
title = {{A survey of software learnability: Metrics, methodologies and guidelines}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {649--658},
url = {http://portal.acm.org/citation.cfm?id=1518803\&dl=ACM},
year = {2009}
}
@inproceedings{VanNiekerk2009,
author = {van Niekerk, J. C. and Roode, J. D.},
title = {{Glaserian and Straussian grounded theory: Similar or completely different?}},
booktitle = {Proc. Conf. South African Institute of Computer Scientists and Information Technologists},
pages = {96--103},
url = {http://portal.acm.org/citation.cfm?id=1632163},
year = {2009}
}
@inproceedings{Andrews2010,
address = {New York, New York, USA},
author = {Andrews, C. and Endert, A. and North, C.},
title = {{Space to think: Large, high-resolution displays for sensemaking}},
doi = {10.1145/1753326.1753336},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {55},
url = {http://portal.acm.org/citation.cfm?doid=1753326.1753336},
year = {2010}
}
@inproceedings{Bernstein2010,
author = {Bernstein, M. S. and Little, G. and Miller, R. C. and Hartmann, B. and Ackerman, M. S. and Karger, D. R. and Crowell, D. and Panovich, K},
title = {{Soylent: A word processor with a crowd inside}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
pages = {313--322},
url = {http://portal.acm.org/citation.cfm?id=1866029.1866078},
year = {2010}
}
@inproceedings{Marshall2005,
author = {Marshall, C. C. and Bly, S.},
title = {{Saving and using encountered information: Implications for electronic periodicals}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {111--120},
url = {http://portal.acm.org/citation.cfm?id=1054989},
year = {2005}
}
@article{Younos2012,
author = {Younos, A.},
title = {{Online education for developing contexts}},
doi = {10.1145/2382856.2382868},
journal = {ACM Crossroads (XRDS)},
number = {2},
pages = {27},
url = {http://dl.acm.org/citation.cfm?doid=2382856.2382868},
volume = {19},
year = {2012}
}
@book{Tufte1997,
title={Visual Explanations: Images and Quantities, Evidence and Narrative},
author={Tufte, E. R.},
year={1997},
publisher={Graphics Press}
}
@article{Liu2010,
author = {Liu, Z. and Stasko, J. T.},
doi = {10.1109/TVCG.2010.177},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {999--1008},
title = {{Mental models, visual reasoning and interaction in information visualization: a top-down perspective}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975137},
volume = {16},
year = {2010}
}
@book{Stephenson1967,
title={The Play Theory of Mass Communication},
author={Stephenson, W.},
year={1967},
publisher={University of Chicago Press}
}
@incollection{Stephenson1967a,
author = {Stephenson, W.},
title = {{Two New Theories of Mass Communication}},
booktitle = {The Play Theory of Mass Communication},
chapter = {1},
pages = {1--12},
year = {1967}
}
@incollection{Stephenson1967b,
author = {Stephenson, W.},
title = {{Play Theory}},
booktitle = {The Play Theory of Mass Communication},
chapter = {4},
pages = {45--65},
year = {1967}
}
@incollection{Stephenson1967c,
author = {Stephenson, W.},
title = {{Ludenic Theory of Newsreading}},
booktitle = {The Play Theory of Mass Communication},
chapter = {11},
pages = {147--159},
year = {1967}
}
@incollection{Stephenson1967d,
author = {Stephenson, W.},
title = {{The Play Theory of Mass Communication Broadly Considered}},
booktitle = {The Play Theory of Mass Communication},
chapter = {15},
pages = {190--206},
year = {1967}
}
@inproceedings{Tweedie1997,
author = {Tweedie, L.},
title = {{Characterizing interactive externalizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/258549.258803},
pages = {375--382},
url = {http://portal.acm.org/citation.cfm?doid=258549.258803},
year = {1997}
}
@incollection{Bederson2003,
author = {Bederson, B. B. and Shneiderman, B.},
title = {{Theories for understanding information visualization}},
booktitle = {The Craft of Information Visualization: Readings and Reflections},
chapter = {8},
pages = {349--351},
publisher = {Morgan-Kaufmann},
year = {2003}
}
@inproceedings{Fisher2012a,
author = {Fisher, D. and Popov, I. and Drucker, S. M. and Schraefel, Mc},
title = {{Trust Me, I'm Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
year = {2012}
}
@inproceedings{Fisher2011,
author = {Fisher, D.},
title = {{Incremental, approximate database queries and uncertainty for exploratory visualization}},
booktitle = {Proc. IEEE Symp. Large Data Analysis and Visualization (LDAV)},
doi = {10.1109/LDAV.2011.6092320},
pages = {73--80},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6092320},
year = {2011}
}
@article{Churchill2013,
author = {Churchill, E. F.},
title = {A matter of taste},
journal = {interactions},
volume = {20},
number = {1},
year = {2013},
pages = {14--17},
url = {http://doi.acm.org/10.1145/2405716.2405721},
doi = {10.1145/2405716.2405721},
}
@article{Friel2001,
author = {Friel, S. N. and Curcio, F. R. and Bright, G. W.},
journal = {Research in Mathematics Education},
number = {2},
pages = {124--158},
title = {{Making sense of graphs: Critical factors influencing comprehension and instructional implications}},
volume = {32},
year = {2001}
}
@book{Spence2007,
author = {Spence, R.},
title = {Information Visualization: Design for Interaction},
edition = {2nd},
publisher = {Prentice Hall},
year = {2007}
}
@book{Wilkinson2005,
author = {Wilkinson, L.},
title = {The Grammar of Graphics},
edition = {2nd},
publisher = {Springer},
year = {2005}
}
@inproceedings{Card1997,
author = {Card, S. K. and Mackinlay, J. D.},
title = {{The structure of the information visualization design space}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {92--99},
year = {1997}
}
@article{Kosara2013,
author = {Kosara, R. and Mackinlay, J. D.},
journal = {IEEE Computer - Special Issue on Cutting-Edge Research in Visualization},
title = {{Storytelling: The next step for visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6412677},
doi = {10.1109/MC.2013.36},
volume = {in press},
year = {2013}
}
@article{Fisher2012b,
author = {Fisher, D. and DeLine, R. and Czerwinski, M. and Drucker, S. M.},
journal = {interactions},
number = {19},
issue = {3},
pages = {50--59},
title = {{Interactions with big data analytics}},
year = {2012}
}
@inproceedings{Buxton1986,
author = {Buxton, W.},
title = {{Chunking and phrasing and the design of human-computer dialogues}},
booktitle = {Proc. IFIP World Computer Congress},
pages = {475--480},
year = {1986}
}
@article{Lee2012a,
author = {Lee, B. and Isenberg, P. and Riche, N. H. and Carpendale, S.},
doi = {10.1109/TVCG.2012.204},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2689--2698},
title = {{Beyond mouse and keyboard: Expanding design considerations for information visualization interactions}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327275},
volume = {18},
year = {2012}
}
@article{Pike2009,
author = {Pike, W. A. and Stasko, J. T. and Chang, R. and O'Connell, T. A.},
doi = {10.1057/ivs.2009.22},
journal = {Information Visualization},
number = {4},
pages = {263--274},
title = {{The science of interaction}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2009.22},
volume = {8},
year = {2009}
}
@article{Klein2006a,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 1: Alternative Perspectives}},
journal = {IEEE Intelligent Systems},
number = {4},
pages = {70--73},
volume = {21},
year = {2006}
}
@article{Klein2006,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 2: A Macrocognitive Model}},
journal = {IEEE Intelligent Systems},
number = {5},
pages = {88--92},
volume = {21},
year = {2006}
}
@inproceedings{Roth2012b,
author = {Roth, R. E.},
title = {{An empirically-derived taxonomy of cartographic interaction primitives}},
booktitle = {Proc. Intl. Conf. Geographic Information Science},
pages = {1--6},
year = {2012}
}
@article{Roth2012,
author = {Roth, R. E.},
journal = {Cartographic Journal},
number = {4},
pages = {376 -- 395},
title = {{Cartographic interaction primitives: Framework and synthesis}},
volume = {49},
year = {2012}
}
@book{Norman1988,
author = {Norman, D. A.},
publisher = {Basic Books New York},
title = {The Psychology of Everyday Things},
year = {1988}
}
@book{Munzner2014,
author = {Munzner, T.},
edition = {Draft},
pages = {1--306},
publisher = {AK Peters},
title = {{Visualization Analysis and Design: Abstractions, Principles, and Methods}},
year = {2014}
}
@incollection{Munzner2009a,
author = {Munzner, T.},
title = {{Visualization}},
booktitle = {Fundamentals of Graphics},
chapter = {27},
edition = {3rd},
editor = {Shirley, P. and Marschner, S.},
pages = {675--707},
publisher = {AK Peters},
year = {2009}
}
@book{Hutchins1995,
author = {Hutchins, E.},
title = {Cognition in the Wild},
publisher = {MIT Press},
year = {1995}
}
@article{Toms2000,
author = {Toms, E. G.},
doi = {10.1006/ijhc.1999.0345},
journal = {Intl. J. Human-Computer Studies},
number = {3},
pages = {423--452},
title = {{Understanding and facilitating the browsing of electronic text}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581999903452},
volume = {52},
year = {2000}
}
@inproceedings{Toms1999,
author = {Toms, E. G.},
title = {{Serendipitous information retrieval}},
booktitle = {Proc. WS. Information Seeking, Searching and Querying in Digital Libraries},
number = {1968},
pages = {1--4},
year = {1999}
}
@article{Micallef2012,
author = {Micallef, L. and Dragicevic, P. and Fekete, J. D.},
doi = {10.1109/TVCG.2012.199},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2536--2545},
title = {{Assessing the effect of visualizations on Bayesian reasoning through crowdsourcing}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327259},
volume = {18},
year = {2012}
}
@book{Case2008,
author = {Case, D. O.},
publisher = {Emerald Publishing Ltd.},
title = {{Looking for Information}},
year = {2008}
}
@article{Kirsh1994,
author = {Kirsh, D. and Maglio, P.},
doi = {10.1207/s15516709cog1804\_1},
journal = {Cognitive Science},
number = {4},
pages = {513--549},
title = {{On distinguishing epistemic from pragmatic action}},
url = {http://doi.wiley.com/10.1207/s15516709cog1804\_1},
volume = {18},
year = {1994}
}
@article{Crouser2012,
author = {Crouser, R. J. and Chang, R.},
doi = {10.1109/TVCG.2012.195},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2859--2868},
title = {{An affordance-based framework for human computation and human-computer collaboration}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327292},
volume = {18},
year = {2012}
}
@article{Kirsh2006,
author = {Kirsh, D.},
doi = {10.1075/pc.14.2.06kir},
journal = {Pragmatics & Cognition},
number = {2},
pages = {249--262},
title = {{Distributed cognition: A methodological note}},
url = {http://openurl.ingenta.com/content/xref?genre=article\&issn=0929-0907\&volume=14\&issue=2\&spage=249},
volume = {14},
year = {2006}
}
@article{Hollan2000,
author = {Hollan, J. and Hutchins, E. and Kirsh, D.},
doi = {10.1145/353485.353487},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {2},
pages = {174--196},
title = {{Distributed cognition: toward a new foundation for human-computer interaction research}},
url = {http://portal.acm.org/citation.cfm?doid=353485.353487},
volume = {7},
year = {2000}
}
@book{Kuhn1962,
author = {Kuhn, T. S.}.,
title = {The Structure of Scientific Revolutions},
publisher = {University of Chicago Press},
edition = {4th},
year = {1962}
}
@article{Liu2008,
author = {Liu, Z. and Nersessian, N. and Stasko, J. T.},
doi = {10.1109/TVCG.2008.121},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1173--80},
title = {{Distributed cognition as a theoretical framework for information visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988961},
volume = {14},
year = {2008}
}
@article{Pohl2012a,
author = {Pohl, M. and Smuc, M. and Mayr, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2908--2916},
title = {{The user puzzle: Explaining the interaction with visual analytics systems}},
volume = {18},
year = {2012}
}
@article{Ma2012,
author = {Ma, J. and Liao, I. and Ma, K. L. and Frazier, J.},
doi = {10.1109/TVCG.2012.244},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2799--2808},
title = {{Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327286},
volume = {18},
year = {2012}
}
@inproceedings{Anand2012,
author = {Anand, A. and Wilkinson, L. and Dang, T. N.},
title = {{Visual pattern discovery using random projections}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {43--52},
year = {2012}
}
@incollection{Rensink2012,
author = {Rensink, R. A.},
pages = {1--24},
title = {{On the prospects for a science of visualization}},
year = {2012}
}
@article{Dou2009,
author = {Dou, W. and Jeong, Do. H. and Stukes, F. and Ribarsky, W. and Lipford, H. R. and Chang, R.},
title = {{Recovering reasoning processes from user interactions}},
doi = {10.1109/MCG.2009.49},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {3},
pages = {52--61},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4909118},
volume = {29},
year = {2009}
}
@inproceedings{Shrinivasan2008,
author = {Shrinivasan, Y. B. and van Wijk, J. J.},
title = {{Supporting the analytical reasoning process in information visualization}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1237--1246},
year = {2008}
}
@article{Tukey1980,
author = {Tukey, J. W.},
journal = {The American Statistician},
number = {1},
pages = {23--25},
title = {{We need both exploratory and confirmatory}},
volume = {34},
year = {1980}
}
@article{Roth2012,
author = {Roth, R. E.},
journal = {Spatial Information Science},
pages = {1--58},
title = {{Interactive mapping: What we know and what we need to know}},
year = {2012}
}
@article{Plaisant1995,
author = {Plaisant, C. and Carr, D. and Shneiderman, B.},
journal = {IEEE Software},
number = {2},
pages = {21--32},
title = {{Image-browser taxonomy and guidelines for designers}},
volume = {12},
year = {1995}
}
@inproceedings{Chuah1996,
author = {Chuah, M. C. and Roth, S. F.},
title = {{On the semantics of interactive visualizations}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.1996.559213},
pages = {29--36},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=559213},
year = {1996}
}
@inproceedings{Tory2004,
author = {Tory, M. and M⬠T.},
title = {{Rethinking visualization: A high-level Taxonomy}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.59},
pages = {151-158},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382903},
year = {2004}
}
@article{Mullins1993,
author = {Mullins, P. M. and Treu, S.},
doi = {http://dx.doi.org/10.1016/0953-5438(93)90016-M},
journal = {Interacting with Computers},
number = {2},
pages = {139--166},
title = {{A task-based cognitive model for user-network interaction: defining a task taxonomy to guide the interface designer}},
volume = {5},
year = {1993}
}
@inproceedings{Gotz2008,
author = {Gotz, D. and Zhou, M. X.},
title = {{Characterizing users' visual analytic activity for insight provenance}},
doi = {10.1109/VAST.2008.4677365},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {123--130},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4677365},
year = {2008}
}
@article{Casner1991,
author = {Casner, S. M.},
journal = {ACM Trans. Graphics (TOG)},
number = {2},
pages = {111--151},
title = {{A Task-Analytic Approach to the Automated Design of Graphic Presentations}},
volume = {10},
year = {1991}
}
@article{Jankun-Kelly2007,
author = {Jankun-Kelly, T. J. and Ma, K. L. and Gertz, M.},
doi = {10.1109/TVCG.2007.28},
title = {{A model and framework for visualization exploration}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {357--69},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17218751},
volume = {13},
year = {2007}
}
@article{Lam2008a,
author = {Lam, H.},
doi = {10.1109/TVCG.2008.109},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{A framework of interaction costs in information visualization}},
number = {6},
pages = {1149--1156},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988958},
volume = {14},
year = {2008}
}
@inproceedings{Lee2006,
author = {Lee, B. and Plaisant, C. and Parr, C. S. and Fekete, J. D. and Henry, N.},
title = {{Task taxonomy for graph visualization categories and subject descriptors}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@article{Morse2000,
author = {Morse, E. and Lewis, M. and Olsen, K. A.},
title = {{Evaluating visualizations: using a taxonomic guide}},
doi = {10.1006/ijhc.2000.0412},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {637--662},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904129},
volume = {53},
year = {2000}
}
@inproceedings{Roth1990,
author = {Roth, S. F. and Mattis, J.},
title = {{Data characterization for intelligent graphics presentation}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/97243.97273},
pages = {193--200},
url = {http://portal.acm.org/citation.cfm?doid=97243.97273},
year = {1990}
}
@article{Amar2005a,
author = {Amar, R. and Stasko, J. T.},
title = {{Knowledge precepts for design and evaluation of information visualizations}},
doi = {10.1109/TVCG.2005.63},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {432--442},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16138553},
volume = {11},
year = {2005}
}
@inproceedings{Wehrend1990,
author = {Wehrend, S. and Lewis, C.},
title = {{A ~Problem-Oriented Classification of Visualization Technique}},
booktitle = {Proc. IEEE Visualization '90},
pages = {139--143},
year = {1990}
}
@inproceedings{Chi1998,
author = {Chi, E. and Riedl, J. T.},
title = {{An operator interaction framework for visualization systems}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {63--70},
year = {1998}
}
@article{Kandel2012,
author = {Kandel, S. and Paepcke, A. and Hellerstein, J. M. and Heer, J.},
doi = {10.1109/TVCG.2012.219},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {2917--2926},
title = {{Enterprise data analysis and visualization: An interview study}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327298},
volume = {18},
year = {2012}
}
@techreport{Sedlmair2012a,
author = {Sedlmair, M. and Brehmer, M. and Ingram, S. and Munzner, T.},
institution = {Dept. Computer Science, University of British Columbia},
title = {{Dimensionality reduction in the wild: Gaps and guidance}},
year = {2012}
}
@inproceedings{Rester2007a,
author = {Rester, M. and Pohl, M. and Wiltner, S. and Hinum, K. and Miksch, S. and Popow, C. and Ohmann, S.},
title = {{Evaluating an InfoVis technique using insight reports}},
booktitle = {Proc. IEEE Intl. Conf. Information Visualization},
pages = {693--700},
year = {2007}
}
@incollection{Munzner2011a,
editor = {Hagen, H.},
author = {Munzner, T and Barsky, A. and Williams, M.},
title = {{Reflections on QuestVis: A visualization system for an environmental sustainability model}},
booktitle = {Scientific Visualization: Interactions, Features, Metaphors. Dagstuhl Follow-Ups},
publisher = {Dagstuhl Publishing},
chapter = {17},
pages = {240--259},
volume = {2},
year = {2011}
}
@inproceedings{Bertini2008,
author = {Bertini, E. and Perer, A. and Plaisant, C. and Santucci, G.},
title = {{Beyond time and errors: Novel evaluation methods for information visualization}},
booktitle = {Proc. WS BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {3913--3916},
year = {2008}
}
@inproceedings{Zhou1998,
author = {Zhou, M. X. and Feiner, S. K.},
title = {{Visual task characterization for automated visual discourse synthesis}},
doi = {10.1145/274644.274698},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {392--399},
url = {http://portal.acm.org/citation.cfm?doid=274644.274698},
year = {1998}
}
@article{Yi2007,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, J. A.},
doi = {10.1109/TVCG.2007.70515},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1224--1231},
title = {{Toward a deeper understanding of the role of interaction in information visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968068},
volume = {13},
year = {2007}
}
@inproceedings{Meyer2012,
author = {Meyer, M. and Sedlmair, M. and Munzner, T.},
title = {{The Four-Level Nested Model Revisited: Blocks and Guidelines}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@article{Heer2012a,
author = {Heer, J. and Shneiderman, B.},
journal = {Communications of the ACM},
pages = {1--26},
title = {{Interactive Dynamics for Visual Analysis A taxonomy of tools that support the fluent and flexible use of visualizations}},
year = {2012}
}
@techreport{Majumder2010,
author = {Majumder, M. and Hofmann, H. and Cook, D.},
institution = {Dept. Statistics, Iowa State University},
pages = {1--6},
title = {{Visual Statistical Inference for Regression Parameters}},
year = {2010}
}
@techreport{Zhao2012,
author = {Zhao, Y. and Cook, D. and Hofmann, H. and Majumder, M. and Chowdhury, N. R.},
institution = {Dept. Statistics, Iowa State University},
pages = {1--25},
title = {{Mind Reading Using an Eyetracker to See How People Are Looking at Lineups}},
year = {2012}
}
@article{Robertson2008,
author = {Robertson, G. G. and Fernandez, R. and Fisher, D. and Lee, B. and Stasko, J. T.},
title = {{Effectiveness of animation in trend visualization}},
doi = {10.1109/TVCG.2008.125},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1325--32},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988980},
volume = {14},
year = {2008}
}
@inproceedings{Pohl2010,
author = {Pohl, M. and Wiltner, S. and Miksch, S.},
title = {{Exploring Information Visualization - Describing Different Interaction Patterns}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Sedlmair2012,
author = {Sedlmair, M. and Tatu, A. and Munzner, T. and Tory, M.},
title = {A Taxonomy of Visual Cluster Separation Factors},
journal ={Computer Graphics Forum (Proc. EuroVis)},
volume = {31},
number = {3},
pages = {1335--1344},
year = {2012}
}
@article{Sedlmair2012b,
author = {Sedlmair, M. and Meyer, M. and Munzner, T.},
title = {{Design Study Methodology: Reflections from the Trenches and the Stacks}},
journal ={IEEE Trans. Visualization and Computer Graphics (TVCG)},
year = {2012},
note = {In press}
}
@article{Wickham2011a,
author = {Wickham, H.},
title = {{Ggplot2}},
doi = {10.1002/wics.147},
journal = {Wiley Interdisciplinary Reviews: Computational Statistics},
number = {2},
pages = {180--185},
url = {http://doi.wiley.com/10.1002/wics.147},
volume = {3},
year = {2011}
}
@article{Buja2009,
author = {Buja, A. and Cook, D. and Hofmann, H. and Lawrence, M. and Lee, E. K. and Swayne, D. F. and Wickham, H.},
doi = {10.1098/rsta.2009.0120},
journal = {Royal Society Philosophical Transactions A},
number = {1906},
pages = {4361--4383},
title = {{Statistical inference for exploratory data analysis and model diagnostics}},
volume = {367},
year = {2009}
}
@article{Wickham2010b,
author = {Wickham, H.},
doi = {10.1198/jcgs.2009.07098},
journal = {Computational and Graphical Statistics},
number = {1},
pages = {3--28},
title = {{A Layered Grammar of Graphics}},
url = {http://pubs.amstat.org/doi/abs/10.1198/jcgs.2009.07098},
volume = {19},
year = {2010}
}
@techreport{Ingram2012,
author = {Ingram, S. and Munzner, T. and Stray, J.},
title = {{Hierarchical Clustering and Tagging of Mostly Disconnected Data}},
institution = {Dept. Computer Science, University of British Columbia},
year = {2012}
}
@inproceedings{Beaudouin-Lafon2004,
author = {Beaudouin-Lafon, M.},
title = {{Designing interaction, not interfaces}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {15--22},
year = {2004}
}
@inproceedings{Kosara2010,
author = {Kosara, R. and Ziemkiewicz, C.},
title = {{Do Mechanical Turks dream of square pie charts?}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Cleveland1984a,
author = {Cleveland, W. S. and McGill, R.},
journal = {J. American Statistical Association},
number = {387},
pages = {531-- 554},
title = {{Graphical perception: Theory, experimentation, and application to the development of graphical methods}},
url = {http://www.jstor.org/stable/2288400},
volume = {79},
year = {1984}
}
@inproceedings{Heer2010,
author = {Heer, J. and Bostock, M.},
title = {{Crowdsourcing graphical perception: using mechanical turk to assess visualization design}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {203--212},
url = {http://dl.acm.org/citation.cfm?id=1753357},
year = {2010}
}
@inproceedings{VanWijk1999,
author = {Wijk, J. J. van and Selow, E. R. van},
title = {{Cluster and calendar based visualization of time series data}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {pages4--9},
year = {1999}
}
@inproceedings{Kairam2012,
author = {Kairam, S. and Maclean, D. and Savva, M. and Heer, J.},
title = {{GraphPrism: Compact visualization of network structure}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
year = {2012}
}
@article{Pinaud2012,
author = {Pinaud, B. and MelanǮ and Dubois, J.},
doi = {10.1111/j.1467-8659.2012.03119.x},
journal = {Computer Graphics Forum},
number = {3pt4},
pages = {1265--1274},
title = {{PORGY: A visual graph rewriting environment for complex systems}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03119.x},
volume = {31},
year = {2012}
}
@inproceedings{Ziemkiewicz2012,
author = {Ziemkiewicz, C. and Gomez, S. and Laidlaw, D. H.},
title = {{Analysis within and between graphs: Observed user strategies in immunobiology visualization}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@article{Wood2011,
author = {Wood, J. and Badawood, D. and Dykes, J. and Slingsby, A.},
doi = {10.1109/TVCG.2011.174},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2384--91},
title = {{BallotMaps: Detecting name bias in alphabetically ordered ballot papers}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034359},
volume = {17},
year = {2011}
}
@article{Heer2007a,
author = {Heer, J. and Vi젆. B. and Wattenberg, M.},
journal = {Communications of the ACM},
number = {1},
pages = {87--97},
title = {{Voyagers and Voyeurs: Supporting Asynchronous Collaborative Visualization}},
volume = {52},
year = {2007}
}
@inproceedings{Wagstaff2012,
author = {Wagstaff, K. L.},
title = {{Machine Learning that Matters}},
booktitle = {Proc. Intl. Conf. Machine Learning},
year = {2012}
}
@article{Marchionini2006,
author = {Marchionini, G.},
journal = {Communications of the ACM},
number = {4},
pages = {41--46},
title = {{Exploratory search: From finding to understanding}},
volume = {49},
year = {2006}
}
@article{Grimmer2011,
author = {Grimmer, J. and King, G.},
doi = {10.1073/pnas.1018067108},
journal = {Proc. National Academy of Sciences of the United States of America},
number = {7},
pages = {2643--50},
title = {{General purpose computer-assisted clustering and conceptualization}},
url = {http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3041127\&tool=pmcentrez\&rendertype=abstract},
volume = {108},
year = {2011}
}
@inproceedings{Budiu2009,
author = {Budiu, R. and Pirolli, P. and Hong, L.},
title = {{Remembrance of Things Tagged: How Tagging Effort Affects Tag Production and Human Memory}},=
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {615--624},
year = {2009}
}
@article{Dasgupta2012,
author = {Dasgupta, A. and Chen, M. and Kosara, R.},
title = {{Conceptualizing Visual Uncertainty in Parallel Coordinates}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
doi = {10.1111/j.1467-8659.2012.03094.x},
pages = {1015--1024},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03094.x},
volume = {31},
year = {2012}
}
@inproceedings{Kandel2012a,
author = {Kandel, S. and Parikh, R. and Paepcke, A. and Hellerstein, J. M. and Heer, J.},
title = {{Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
year = {2012}
}
@inproceedings{Heer2012,
author = {Heer, J. and Stone, M.},
title = {{Color naming models for color selection, image editing and palette design}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208547},
pages = {1007},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208547},
year = {2012}
}
@techreport{Brandes2012,
author = {Brandes, U. and Nick, B. and Rockstroh, B. and Steffen, A.},
title = {{Gestaltlines}},
institution = {Dept. Computer and Information Science, University of Konstanz},
year = {2012}
}
@inproceedings{Willett2012,
author = {Willett, W. and Heer, J. and Agrawala, M.},
title = {{Strategies for Crowdsourcing Social Data Analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Jianu2012,
author = {Jianu, R. and Laidlaw, D.},
title = {{An Evaluation of How Small User Interface Changes Can Improve Scientists' Analytic Strategies}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2953--2962},
year = {2012}
}
@inproceedings{Eisenstein2012,
author = {Eisenstein, J. and Chau, D. H. P. and Kittur, A. and Xing, E. P.},
title = {{TopicViz: Interactive Topic Exploration in Document Collections}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2177--2182},
year = {2012}
}
@article{Krzywinski2011,
author = {Krzywinski, M. and Birol, I. and Jones, S. M. and Marra, M. A.},
doi = {10.1093/bib/bbr069},
journal = {Briefings in Bioinformatics},
title = {{Hive plots--rational approach to visualizing networks}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22155641},
year = {2011}
}
@inproceedings{Furniss2011,
author = {Furniss, D. and Blandford, A. and Curzon, P. and Mary, Q.},
title = {{Confessions from a grounded theory PhD: experiences and lessons learnt}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {113--122},
url = {http://dl.acm.org/citation.cfm?id=1978960},
year = {2011}
}
@inproceedings{Shipman1999,
author = {Shipman, F. M. and Marshall, C. C.},
title = {{Formality considered harmful: Experiences, emerging themes, and directions on the use of formal representations in interactive systems}},
booktitle = {Proc. ACM Conf. Computer Supported Cooperative Work (CSCW)},
number = {4},
pages = {333--352},
url = {http://www.springerlink.com/index/P7217382837312PM.pdf},
volume = {8},
year = {1999}
}
@article{vanderMaaten2011,
author = {Maaten, L. van der and Hinton, G.},
doi = {10.1007/s10994-011-5273-4},
journal = {Machine Learning},
title = {{Visualizing non-metric similarities in multiple maps}},
url = {http://www.springerlink.com/index/10.1007/s10994-011-5273-4},
year = {2011}
}
@article{Saraiya2005b,
author = {Saraiya, P. and North, C. and Duca, K.},
doi = {10.1109/TVCG.2005.53},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {443--456},
title = {{An insight-based methodology for evaluating bioinformatics visualizations}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16138554},
volume = {11},
year = {2005}
}
@article{Hayes2011,
author = {Hayes, Gillian R.},
doi = {10.1145/1993060.1993065},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {3},
pages = {1--20},
title = {{The relationship of action research to human-computer interaction}},
url = {http://dl.acm.org/citation.cfm?doid=1993060.1993065},
volume = {18},
year = {2011}
}
@article{O'Brien2010,
author = {O'Brien, T. M. and Ritz, A. M. and Raphael, B. J. and Laidlaw, D. H.},
doi = {10.1109/TVCG.2010.163},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {918--26},
title = {{Gremlin: an interactive visualization model for analyzing genomic rearrangements}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975128},
volume = {16},
year = {2010}
}
@inproceedings{Endert2012,
author = {Endert, A. and Fiaux, P. and North, C.},
title = {{Semantic Interaction for Visual Text Analytics}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@article{North2011,
author = {North, C. and Saraiya, P. and Duca, K.},
doi = {10.1177/1473871611415989},
journal = {Information Visualization},
number = {3},
pages = {162--181},
title = {{A comparison of benchmark task and insight evaluation methods for information visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415989},
volume = {10},
year = {2011}
}
@incollection{McGrath1995,
address = {San Fransisco, CA, USA},
author = {McGrath, J. E.},
title = {{Methodology matters: Doing research in the behavioral and social sciences}},
booktitle = {Readings in human-computer interaction: Towards the Year 2000},
edition = {2nd},
editor = {Baecker, R. M. and Grudin, J. and Buxton, W. and Greenberg, S.},
pages = {152 -- 169},
publisher = {Morgan Kaufmann},
url = {http://www.citeulike.org/group/258/article/850344},
year = {1995}
}
@article{Sprague2009,
author = {Sprague, D. and Tory, M.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {4},
pages = {86--91},
title = {{Motivation and procrastination: methods for evaluating pragmatic casual information visualizations}},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=5167491},
volume = {29},
year = {2009}
}
@techreport{Anderson2013,
author = {Anderson, C. W. and Kreiss, D.},
title = {{Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism}},
institution = {Dept. Media Culture, College of Staten Island (CUNY)},
year = {2013}
}
@article{Sprague2012,
author = {Sprague, D. and Tory, M.},
doi = {10.1177/1473871611433710},
journal = {Information Visualization},
number = {2},
pages = {106--123},
title = {{Exploring how and why people use visualizations in casual contexts: Modeling user goals and regulated motivations}},
volume = {11},
year = {2012}
}
@book{Frankfurt2005,
author = {Frankfurt, H. G.},
title = {On Bullshit},
publisher = {Princeton University Press},
year = {2005}
}
@inproceedings{Shneiderman1996,
author = {Shneiderman, B.},
title = {{The eyes have it: a task by data type taxonomy for information visualizations}},
booktitle = {Proc. IEEE Symp. Visual Languages},
doi = {10.1109/VL.1996.545307},
pages = {336--343},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=545307},
year = {1996}
}
@inproceedings{Amar2005,
author = {Amar, R. and Eagan, J. and Stasko, J. T.},
title = {{Low-level components of analytic activity in information visualization}},
doi = {10.1109/INFVIS.2005.1532136},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {111--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532136},
year = {2005}
}
@book{Charmaz2006,
author = {Charmaz, K.},
title = {Constructing Grounded Theory: A Practical Guide through Qualitative Analysis},
url = {http://www.amazon.com/Constructing-Grounded-Theory-Qualitative-Introducing/dp/0761973532},
publisher = {Sage},
year = {2006}
}
@book{Trochim1999,
author = {Trochim, W.},
title = {The Research Methods Knowledge Base},
edition = {2nd},
publisher = {Cornell Custom Publishing},
year = {1999}
}
@article{Ryan2003,
author = {Ryan, G. W. and Bernard, H. R.},
doi = {10.1177/1525822X02239569},
journal = {Field Methods},
number = {1},
pages = {85--109},
title = {{Techniques to Identify Themes}},
url = {http://fmx.sagepub.com/cgi/doi/10.1177/1525822X02239569},
volume = {15},
year = {2003}
}
@article{Mathison1988,
author = {Mathison, S.},
title = {{Why Triangulate?}},
doi = {10.2307/1174583},
journal = {Educational Researcher},
number = {2},
pages = {13},
url = {http://links.jstor.org/sici?sici=0013-189X\%28198803\%2917\%3A2\%3C13\%3AWT\%3E2.0.CO\%3B2-L\&origin=crossref},
volume = {17},
year = {1988}
}
@book{Massironi2001,
author = {Massironi, M.},
publisher = {Psychology Press},
title = {{The Psychology of Graphic Images}},
year = {2001}
}
@article{Krueger2002,
author = {Krueger, R. A.},
title = {{Designing and Conducting Focus Group Interviews}},
year = {2002}
}
@incollection{Fontana1994,
author = {Fontana, A. and Frey, J.},
title = {{Interviewing: The Art of Science}},
booktitle = {The Handbook of Qualitative Research},
edition = {3rd},
pages = {361--376},
publisher = {Sage},
year = {1994}
}
@article{Ratcliff,
author = {Ratcliff, D.E.},
title = {{Analytic induction as a qualitative research method of analysis}},
url = {http://qualitativeresearch.ratcliffs.net/analytic.pdf}
}
@article{Becker1957,
author = {Becker, H. S. and Geer, B.},
journal = {Human Organization},
number = {3},
pages = {28--32},
title = {{Field Methods and Techniques Participant Observation and Interviewing: A Comparison}},
volume = {16},
year = {1957}
}
@article{Newell1985,
author = {Newell, A. and Card, S. K.},
title = {{The prospects for psychological science in human-computer interaction}},
journal = {Human-Computer Interaction},
pages = {209--242},
url = {http://portal.acm.org/citation.cfm?id=1453662},
volume = {1},
year = {1985}
}
@article{Tri-Council2010,
author = {CIHR, NSERC, SSHRC},
title = {{Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans}},
year = {2010}
}
@book{Silverman2007,
author = {Silverman, D.},
publisher = {Sage},
title = {{A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research}},
year = {2007}
}
@article{Wickham2010,
author = {Wickham, H. and Cook, D. and Hofmann, H. and Buja, A.},
title = {{Graphical Inference for Infovis}},
doi = {10.1109/TVCG.2010.161},
issn = {1077-2626},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {973--9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975134},
volume = {16},
year = {2010}
}
@article{Rubinov2006,
author = {Rubinov, A. M. and Soukhorokova, N. V. and Ugon, J.},
doi = {10.1016/j.ejor.2005.04.047},
journal = {European J. Operational Research},
number = {3},
pages = {849--865},
title = {{Classes and clusters in data analysis}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0377221705006867},
volume = {173},
year = {2006}
}
@article{Wickham2011,
author = {Wickham, H. and Hofmann, H.},
doi = {10.1109/TVCG.2011.227},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2223--30},
title = {{Product plots}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034341},
volume = {17},
year = {2011}
}
@book{Crotty1998,
author = {Crotty, M.},
title = {{The Foundations of Social Research}},
publisher = {Sage},
year = {1998}
}
@article{Rodgers2011,
author = {Rodgers, J. and Bartram, L.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2489--2497},
title = {{Exploring Ambient and Artistic Visualization for Residential Energy Use Feedback}},
volume = {17},
year = {2011}
}
@article{Matusik2003,
author = {Matusik, W. and Pfister, H. and Brand, M. and McMillan, L.},
title = {{A data-driven reflectance model}},
doi = {10.1145/882262.882343},
journal = {ACM Trans. Graphics},
number = {3},
pages = {759},
url = {http://portal.acm.org/citation.cfm?doid=882262.882343},
volume = {22},
year = {2003}
}
@incollection{Peterman2009,
author = {Peterman, R. M.},
booktitle = {The Future of Fisheries Science in North America},
pages = {167--184},
title = {{Fisheries Science in the Future}},
year = {2009}
}
@book{Gigerenzer2007,
author = {Gigerenzer, G.},
title = {{Gut Feelings: The Intelligence of the Unconscious}},
publisher = {Viking},
year = {2007}
}
@inproceedings{Kang2011,
author = {Kang, Y. A. and Stasko, J. T.},
title = {{Characterizing the Intelligence Analysis Process: Informing Visual Analytics Design through a Longitudinal Field Study}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {19--28},
volume = {17},
year = {2011}
}
@article{Chang2009,
author = {Chang, R. and Ziemkiewicz, C. and Green, T. M. and Ribarsky, W.},
title = {{Defining insight for visual analytics}},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {2},
pages = {14--7},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19462629},
volume = {29},
year = {2009}
}
@article{Klahr1999,
author = {Klahr, D. and Simon, H. A.},
journal = {Psychological Bulletin},
number = {5},
pages = {524--543},
title = {{Studies of Scientific Creativity: Complimentary approaches and convergent findings}},
volume = {125},
year = {1999}
}
@article{Ingram2009,
author = {Ingram, S. and Munzner, T. and Olano, M.},
title = {{Glimmer: multilevel MDS on the GPU}},
doi = {10.1109/TVCG.2008.85},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {249--61},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19147889},
volume = {15},
year = {2009}
}
@article{North2006,
author = {North, C.},
title = {{Toward measuring visualization insight}},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {3},
pages = {6-9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16711210},
volume = {26},
year = {2006}
}
@inproceedings{Jennings2011,
author = {Jennings, K. E. and Simonton, D. K. and Palmer, S. E.},
title = {{Understanding Exploratory Creativity in a Visual Domain}},
booktitle = {Proc. ACM Conf. Creativity and Cognition},
number = {1},
year = {2011}
}
@inproceedings{Nardi2000,
author = {Nardi, B. A. and Whittaker, S. and Bradner, E.},
title = {{Interaction and Outeraction: Instant Messaging in Action}},
booktitle = {Proc. ACM Conf. Computer Supported Cooperative Work (CSCW)},
pages = {79--88},
year = {2000}
}
@inproceedings{Kraut1988,
author = {Kraut, R. and Egido, C. and Galegher, J.},
title = {{Patterns of Contact and Communication Collaboration in Scientific Research Collaboration}},
booktitle = {Proc. ACM Conf. Computer Supported Cooperative Work (CSCW)},
pages = {1--12},
year = {1988}
}
@inproceedings{Andre2009,
author = {Andre, P. and Teevan, J. and Dumais, S. T.},
title = {{Discovery is never by chance: Designing for (un)serendipity}},
booktitle = {Proc. ACM Conf. Creativity and Cognition},
pages = {305--314},
year = {2009}
}
@inproceedings{Findlater2009a,
author = {Findlater, L. and Moffatt, K. A. and McGrenere, J. and Dawson, J.},
title = {{Ephemeral adaptation: The use of gradual onset to improve menu selection performance}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1655--1664},
url = {http://portal.acm.org/citation.cfm?id=1518701.1518956},
year = {2009}
}
@article{Munzner2009,
author = {Munzner, T.},
doi = {10.1109/TVCG.2009.111},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {921--8},
title = {{A nested model for visualization design and validation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834155},
volume = {15},
year = {2009}
}
@article{Hullman2011a,
author = {Hullman, J. and Adar, E. and Shah, P.},
doi = {10.1109/TVCG.2011.175},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2213--2222},
title = {{Benefitting InfoVis with Visual Difficulties}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034340},
volume = {17},
year = {2011}
}
@inproceedings{Springmeyer1992,
author = {Springmeyer, R. R. and Blattner, M. M. and Max, N. L.},
title = {{A characterization of the scientific data analysis process}},
booktitle = {Proc. IEEE Visualization},
doi = {10.1109/VISUAL.1992.235203},
pages = {235--242},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=235203},
year = {1992}
}
@book{Card1999,
author = {Card, S. K. and Mackinlay, J. D. and Schneiderman, B.},
publisher = {Morgan Kaufmann},
title = {{Readings in information: visualization using vision to think}},
url = {http://cat.inist.fr/?aModele=afficheN\&cpsidt=110800},
year = {1999}
}
@inproceedings{Wilkinson2005,
author = {Wilkinson, L. and Anand, A. and Grossman, R.},
title = {{Graph-theoretic scagnostics}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
publisher = {Citeseer},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.6148\&rep=rep1\&type=pdf},
year = {2005}
}
@article{Becker1996a,
author = {Becker, R. A and Cleveland, W. S. and Shyu, M. J.},
doi = {10.2307/1390777},
journal = {Computational and Graphical Statistics},
number = {2},
pages = {123},
title = {{The Visual Design and Control of Trellis Display}},
url = {http://www.jstor.org/stable/1390777?origin=crossref},
volume = {5},
year = {1996}
}
@article{Rensink2010,
author = {Rensink, R. A. and Baldridge, G.},
title = {{The Perception of Correlation in Scatterplots}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {1203--1210},
url = {http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8659.2009.01694.x/full},
volume = {29},
year = {2010}
}
@article{Bertini2011,
author = {Bertini, E. and Tatu, A. and Keim, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2203--2212},
title = {{Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization}},
volume = {17},
year = {2011}
}
@article{Tenenbaum2000,
author = {Tenenbaum, J. B. and de Silva, V. and Langford, J .C.},
title = {{A global geometric framework for nonlinear dimensionality reduction.}},
doi = {10.1126/science.290.5500.2319},
journal = {Science},
number = {5500},
pages = {2319--2323},
url = {http://www.ncbi.nlm.nih.gov/pubmed/11125149},
volume = {290},
year = {2000}
}
@inproceedings{Ingram2010,
author = {Ingram, S. and Munzner, T. and Irvine, V. and Tory, M. and Bergner, S. and Torsten, M.},
title = {{DimStiller: Workflows for dimensional analysis and reduction}},
booktitle = {Proc. IEEE Symp. Visual Analytics Software and Technologies (VAST)},
pages = {3--10},
year = {2010}
}
@incollection{Munzner2011,
author = {Munzner, T.},
title = {{Attribute Reduction Methods}},
booktitle = {Information Visualization: Principles, Methods, and Practice},
chapter = {8},
edition = {Draft},
year = {2011}
}
@book{Tan2006,
address = {Boston, Massachusetts, USA},
author = {Tan, P. N. and Steinbach, M. and Kumar, V.},
pages = {769},
publisher = {Pearson Education},
title = {{Introduction to Data Mining}},
year = {2006}
}
@incollection{Holbrey2006,
author = {Holbrey, R.},
title = {{Column Reduction Techniques}},
booktitle = {Dimension Reduction Algorithms for Data Mining and Visualization},
chapter = {4},
doi = {10.1.1.124.5567},
editor = {Holbrey, R.},
pages = {31-51},
publisher = {Citeseer},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.5567\&rep=rep1\&type=pdf},
year = {2006}
}
@article{Buja2002,
author = {Buja, A. and Swayne, D. F.},
doi = {10.1007/s00357-001-0031-0},
journal = {J. Classification},
pages = {7--43},
title = {{Visualization Methodology for Multidimensional Scaling}},
volume = {19},
year = {2002}
}
@article{vanWijk2006,
author = {Wijk, J. J. van},
title = {{Views on Visualization}},
doi = {10.1109/TVCG.2006.80},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {421--32},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16805253},
volume = {12},
year = {2006}
}
@inproceedings{Wobbrock2011,
author = {Wobbrock, J. O. and Findlater, L. and Gergle, D. and Higgins, J. J.},
title = {{The aligned rank transform for nonparametric factorial analyses using only ANOVA procedures}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {143--146},
year = {2011}
}
@article{Borkin2011,
author = {Borkin, M. A. and Gajos, K. Z. and Peters, A. and Mitsouras, D. and Melchionna, S. and Rybicki, F. J. and Feldman, C. L. and Pfister, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2479--2488},
title = {{Evaluation of Artery Visualizations for Heart Disease Diagnosis}},
volume = {17},
year = {2011}
}
@article{Steinberger2011,
author = {Steinberger, M. and Waldner, M. and Streit, M. and Lex, A. and Schmalstieg, D.},
doi = {10.1109/TVCG.2011.183},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2249--58},
title = {{Context-preserving visual links}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034344},
volume = {17},
year = {2011}
}
@inproceedings{Winckler2004,
author = {Winckler, M. A. and Palanque, P. and Freitas, C. M. D. S.},
title = {{Tasks and scenario-based evaluation of information visualization techniques}},
booktitle = {Proc. Intl. WS. Task Models and Diagrams for User Interface Design (TAMODIA '04)},
doi = {10.1145/1045446.1045475},
url = {http://portal.acm.org/citation.cfm?doid=1045446.1045475},
year = {2004}
}
@inproceedings{Crabtree2009,
author = {Crabtree, A. and Rodden, T. and Tolmie, P. and Button, G.},
title = {{Ethnography considered harmful}},
url = {http://portal.acm.org/citation.cfm?doid=1518701.1518835},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1518701.1518835},
pages = {879--888},
year = {2009}
}
@inproceedings{Zimmerman2010,
author = {Zimmerman, J. and Stolterman, E. and Forlizzi, J.},
title = {{An analysis and critique of Research through Design: towards a formalization of a research approach}},
booktitle = {Proc. ACM Conf. Designing Interactive Systems (DIS '10)},
pages = {310--319},
url = {http://portal.acm.org/citation.cfm?id=1858228},
year = {2010}
}
@article{Greenberg2008,
title = {{Usability evaluation considered harmful (some of the time)}},
author = {Greenberg, S. and Buxton, W.},
doi = {10.1145/1357054.1357074},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {111},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357074},
year = {2008}
}
@article{Lloyd2011,
author = {Lloyd, D. and Dykes, J.},
title = {{Human-centered approaches in geovisualization design: investigating multiple methods through a long-term case study}},
doi = {10.1109/TVCG.2011.209},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {12},
pages = {2498--2507},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034371},
volume = {17},
year = {2011}
}
@book{Pirolli2009,
address = {New York, NY, USA.},
author = {Pirolli, P.},
publisher = {Oxford University Press},
title = {{Information Foraging Theory: Adaptive Interaction with Information}},
year = {2009}
}
@article{Tory2005,
author = {Tory, M. and Moller, T.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {5},
pages = {8--11},
title = {{Evaluating visualizations: do expert reviews work?}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16209163},
volume = {25},
year = {2005}
}
@inproceedings{Kang2010,
author = {Kang, Y. A. and Gorg, C. and Stasko, J.},
title = {{Pragmatic Challenges in the Evaluation of Interactive Visualization Systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Chang2010,
author = {Chang, R. and Ziemkiewicz, C. and Pyzh, R. and Kielman, J. and Ribarsky, W.},
title = {{Learning-Based Evaluation of Visual Analytic Systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Elmqvist2010,
author = {Elmqvist, N.},
title = {{Mutually Linked Studies - Balancing Threats to Internal and Ecological Validity in InfoVis Evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {2--3},
year = {2010}
}
@inproceedings{Tory2008,
author = {Tory, M. and Staub-French, S.},
title = {{Qualitative Analysis of Visualization: A Building Design Field Study}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2008}
}
@inproceedings{Faisal2008,
author = {Faisal, S. and Craft, B. and Cairns, P. and Blandford, A.},
title = {{Internalization, Qualitative Methods, and Evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--8},
year = {2008}
}
@inproceedings{Yi2008,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, J. A},
title = {{Understanding and Characterizing Insights: How Do People Gain Insights Using Information Visualization?}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--6},
year = {2008}
}
@inproceedings{Valiati2008,
author = {Valiati, E. R. A. and Freitas, C. M. D. S. and Pimenta, M. S.},
title = {{Using Multi-dimensional In-depth Long-term Case Studies for information visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1--7},
year = {2008}
}
@book{Ware2004,
address = {San Fransisco, CA, USA},
author = {Ware, C.},
edition = {2},
pages = {228--237},
publisher = {Morgan Kaufmann},
title = {{Information Visualization: Perception for Design}},
year = {2004}
}
@book{Tufte2001,
address = {Cheshire, CT, USA},
author = {Tufte, E. R},
publisher = {Graphics Press},
title = {{The Visual Display of Quantitative Information}},
year = {2001}
}
@book{Tufte1990,
address = {Cheshire, CT, USA},
author = {Tufte, E. R},
publisher = {Graphics Press},
title = {{Envisioning Information}},
year = {1990}
}
@book{Dix2004,
author = {Dix, A. and Finlay, J. and Abowd, G. D. and Beale, R.},
edition = {3rd Edition},
publisher = {Pearson Education Limited},
title = {{Human-Computer Interaction}},
year = {2004}
}
@inproceedings{Zuk2006,
author = {Zuk, T. and Schlesier, L. and Neumann, P. and Hancock, M. S. and Carpendale, S.},
title = {{Heuristics for information visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168162},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168162},
year = {2006}
}
@article{Sutcliffe2000,
author = {Sutcliffe, A. G. and Ennis, M. and Hu, J.},
title = {{Evaluating the effectiveness of visual user interfaces for information retrieval}},
journal = {Intl. J. Human-Computer Interaction},
number = {5},
pages = {741--763},
url = {http://www.sciencedirect.com/science/article/pii/S1071581900904166},
volume = {53},
year = {2000}
}
@book{Thomas2005,
author = {Thomas, J. J. and Cook, K. A.},
publisher = {IEEE},
title = {{Illuminating the path: The research and development agenda for visual analytics}},
url = {http://www.mendeley.com/research/illuminating-path-research-development-agenda-visual-analytics-2/},
volume = {54},
year = {2005}
}
@article{Chen2000a,
author = {Chen, C. and Czerwinski, M.},
title = {{Empirical evaluation of information visualizations: an introduction}},
doi = {10.1006/ijhc.2000.0421},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {631-635},
url = {http://linkinghub.elsevier.com/retrieve/pii/S107158190090421X},
volume = {53},
year = {2000}
}
@article{Perer2009,
author = {Perer, A. and Shneiderman, B.},
title = {{Integrating statistics and visualization for exploratory power: from long-term case studies to design guidelines}},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {3},
pages = {39-51},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19642614},
volume = {29},
year = {2009}
}
@inproceedings{Valiati2006,
author = {Valiati, E. R. A. and Pimenta, M. S. and Freitas, C. M. D. S.},
title = {{A taxonomy of tasks for guiding the evaluation of multidimensional visualizations}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {1-6},
url = {http://portal.acm.org/citation.cfm?id=1168169},
year = {2006}
}
@inproceedings{Gonzalez2003a,
author = {Gonzalez, V. M. and Kobsa, A.},
title = {{A workplace study of the adoption of information visualization systems}},
booktitle = {Proc. Intl. Conf. Knowledge Management (I-KNOW '03)},
pages = {92--102},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.74.3805\&rep=rep1\&type=pdf},
year = {2003}
}
@inproceedings{Mayr2010,
author = {Mayr, E. and Smuc, M. and Risku, H.},
title = {{Many roads lead to Rome: Mapping users' problem-solving strategies}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@inproceedings{Saraiya2010,
author = {Saraiya, P. and North, C. and Duca, K.},
title = {{Comparing Benchmark Task and Insight Evaluation Methods on Timeseries Graph Visualizations}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Saraiya2006,
author = {Saraiya, P. and North, C. and Lam, V. and Duca, K.},
doi = {10.1109/TVCG.2006.85},
title = {{An insight-based longitudinal study of visual analytics}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1511--22},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17073373},
volume = {12},
year = {2006}
}
@inproceedings{Isenberg2008a,
author = {Isenberg, P. and Tang, A. and Carpendale, S.},
title = {{An exploratory study of visual information analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Cystems (CHI)},
doi = {10.1145/1357054.1357245},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357245},
year = {2008}
}
@inproceedings{Plaisant2004,
author = {Plaisant, C.},
title = {{The challenge of information visualization evaluation}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/989863.989880},
url = {http://portal.acm.org/citation.cfm?doid=989863.989880},
year = {2004}
}
@article{Trafton2000,
author = {Trafton, J. G. and Kirschenbaum, S. S. and Tsui, T. L. and Miyamoto, R. T. and Ballas, J. A. and Raymond, P. D.},
doi = {10.1006/ijhc.2000.0419},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {827-850},
title = {{Turning pictures into numbers: extracting and generating information from complex visualizations}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904191},
volume = {53},
year = {2000}
}
@article{Seo2006,
author = {Seo, J. and Shneiderman, B.},
doi = {10.1109/TVCG.2006.50},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {311-22},
title = {{Knowledge discovery in high-dimensional data: case studies and a user survey for the rank-by-feature framework}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16640245},
volume = {12},
year = {2006}
}
@inproceedings{Scholtz2010,
author = {Scholtz, J.},
title = {{Developing Qualitative Metrics for Visual Analytic Environments}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Sedlmair2011,
title = {{Information Visualization Evaluation in Large Companies: Challenges, Experiences and Recommendations}},
author = {Sedlmair, M. and Isenberg, P. and Baur, D. and Butz, A.},
doi = {doi:10.1177/1473871611413099},
journal = {Information Visualization},
pages = {248--266},
url = {http://www.cs.ubc.ca/~msedl/papers/sedlmair2011ivs.pdf},
volume = {10},
year = {2011}
}
@inproceedings{Pirolli2005,
author = {Pirolli, P. and Card, S.},
title = {{The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis}},
booktitle = {Proc. Intl. Conf. Intelligence Analysis},
pages = {2--4},
url = {http://scholar.google.com/scholar?hl=en\&btnG=Search\&q=intitle:The+Sensemaking+Process+and+Leverage+Points+for+Analyst+Technology+as+Identified+Through+Cognitive+Task+Analysis\#0},
volume = {2005},
year = {2005}
}
@inproceedings{Isenberg2008,
author = {Isenberg, P. and Zuk, T. and Collins, C. and Carpendale, S.},
title = {{Grounded evaluation of information visualizations}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1377966.1377974},
url = {http://portal.acm.org/citation.cfm?doid=1377966.1377974},
year = {2008}
}
@inproceedings{Ellis2006,
author = {Ellis, G. and Dix, A.},
title = {{An explorative analysis of user evaluation studies in information visualisation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168152},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168152},
year = {2006}
}
@inproceedings{Andrews2008,
author = {Andrews, K.},
title = {{Evaluation Comes in Many Guises}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
pages = {7-8},
year = {2008}
}
@article{Lam2010,
author = {Lam, H. and Munzner, T.},
journal = {Synthesis Lectures on Visualization},
number = {1},
pages = {1-117},
publisher = {Morgan and Claypool},
title = {{A Guide to Visual Multi-level Interface Design from Synthesis of Empirical Study Evidence}},
url = {http://www.morganclaypool.com/doi/abs/10.2200/S00308ED1V01Y201011VIS001},
volume = {1},
year = {2010}
}
@inproceedings{Shneiderman2006,
author = {Shneiderman, B. and Plaisant, C.},
title = {{Strategies for evaluating information visualization tools: multi-dimensional in-depth long-term case studies}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
doi = {10.1145/1168149.1168158},
url = {http://portal.acm.org/citation.cfm?doid=1168149.1168158},
year = {2006}
}
@incollection{Carpendale2008,
address = {Berlin},
author = {Carpendale, S.},
title = {{Evaluating information visualizations}},
booktitle = {Information Visualization: Information Visualization: Human-Centered Issues and Perspectives},
chapter = {2},
doi = {10.1007/978-3-540-70956-5\_2},
editor = {Kerren, A. and Stasko, J. T. and Fekete, J. D. and North, C.},
pages = {19-45},
publisher = {Springer},
url = {http://www.springerlink.com/index/07U623325P7670R8.pdf},
year = {2008}
}
@inproceedings{Gonzalez2003,
author = {Gonzalez, V. M. and Kobsa, A.},
title = {{Benefits of information visualization systems for administrative data analysts}},
booktitle = {Proc. Intl. Conf. Information Visualization (IV)},
doi = {10.1109/IV.2003.1217999},
pages = {331-336},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1217999},
year = {2003}
}
@article{Lam2011,
author = {Lam, H. and Bertini, E. and Isenberg, P. and Plaisant, C. and Carpendale, S.},
doi = {10.1109/TVCG.2011.279},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
pages = {1--18},
title = {{Empirical Studies in Information Visualization: Seven Scenarios}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22144529},
year = {2011}
}
@inproceedings{Saraiya2004,
author = {Saraiya, P. and North, C. and Duca, K.},
title = {{An Evaluation of Microarray Visualization Tools for Biological Insight}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.5},
pages = {1-8},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382884},
year = {2004}
}
@inproceedings{Amar2004a,
author = {Amar, R. and Stasko, J.},
title = {{A Knowledge task-based framework for design and evaluation of information visualizations}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {143� url = {http://doi.ieeecomputersociety.org/10.1109/INFVIS.2004.10}
year = {2004}
}
}}}
{{{
@inproceedings{Welsh2001,
author = {Welsh, M. and Culler, D. and Brewer, E.},
title = {{SEDA: An architecture for well-conditioned, scalable internet services}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {230--243},
year = {2001}
}
@inproceedings{Yip2009,
author = {Yip, A. and Wang, X. and Zeldovich, N. and Kaashoek, M. F.},
title = {{Improving application security with data flow assertions}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
doi = {10.1145/1629575.1629604},
pages = {291--304},
url = {http://portal.acm.org/citation.cfm?doid=1629575.1629604},
year = {2009}
}
@article{Chase1994,
author = {Chase, J. S. and Levy, H. M. and Feeley, M. J. and Lazowska, E. D.},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {4},
pages = {271--307},
title = {{Sharing and protection in a single-address-space operating system}},
volume = {12},
year = {1994}
}
@inproceedings{Bershad1995,
author = {Bershad, B. and Savage, S. and Pardyak, P. and Sirer, E. and Fuczynski, M. and Becker, D. and Chambers, C. and Eggers, S.},
title = {{Extensibility, safety, and performance in the SPIN operating system}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {267--284},
year = {1995}
}
@inproceedings{Liedtke1995,
author = {Liedtke, J.},
title = {{On micro-kernel construction}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {237--250},
year = {1995}
}
@inproceedings{Swift2003,
author = {Swift, M. M. and Bershad, B. N. and Levy, H. M.},
title = {{Improving the reliability of commodity operating systems}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
doi = {10.1145/945465.945466},
pages = {207--222},
url = {http://portal.acm.org/citation.cfm?doid=945445.945466},
year = {2003}
}
@inproceedings{Barham2003,
author = {Barham, P. and Dragovic, B. and Fraser, K. and Hand, S. and Harris, T. and Ho, A. and Neugebauer, R. and Pratt, I. and Warfield, A.},
title = {{Xen and the art of virtualization}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {164--177},
year = {2003}
}
@inproceedings{Bugnion1997,
author = {Bugnion, E. and Devine, S. and Rosenblum, M.},
title = {{Disco: Running commodity operating systems on scalable multiprocessors}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {143--156},
year = {1997}
}
@article{Pai2000,
author = {Pai, V. S. and Druschel, P. and Zwaenepoel, W.},
doi = {10.1145/332799.332895},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {1},
pages = {37--66},
title = {{IO-Lite: A unified I/O buffering and caching system}},
url = {http://portal.acm.org/citation.cfm?doid=332799.332895},
volume = {18},
year = {2000}
}
@article{Ghemawat2003,
author = {Ghemawat, S. and Gobioff, H. and Leung, S. T.},
doi = {10.1145/1165389.945450},
journal = {ACM SIGOPS Operating Systems Review},
number = {5},
pages = {29--43},
title = {{The Google file system}},
url = {http://portal.acm.org/citation.cfm?doid=1165389.945450},
volume = {37},
year = {2003}
}
@inproceedings{Efstathopoulos2005,
author = {Efstathopoulos, P. and Krohn, M. and Vandebogart, S. and Frey, C. and Ziegler, D. and Kohler, E. and Mazi전. and Kaashoek, M. F. and Morris, R.},
title = {{Labels and event processes in the Asbestos operating system}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {17--30},
year = {2005}
}
@inproceedings{Mosberger1996,
author = {Mosberger, D. and Peterson, L. L.},
title = {{Making paths explicit in the Scout operating system}},
booktitle = {Proc. USENIX Symp. Operating Systems Design and Implementation (OSDI)},
pages = {153--168},
year = {1996}
}
@inproceedings{Klein2009,
author = {Klein, G. and Elphinstone, K. and Heiser, G. and Engelhardt, K.},
title = {{seL4: Formal verification of an OS kernel}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {207--220},
year = {2009}
}
@inproceedings{Behren2003,
author = {Behren, R. von and Condit, J. and Zhou, F. and Necula, G. C. and Brewer, E.},
title = {{Capriccio: Scalable threads for internet services}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {268--281},
year = {2003}
}
@article{Saltzer1974,
author = {Saltzer, J. H.},
doi = {10.1145/361011.361067},
journal = {Communications of the ACM},
number = {7},
pages = {388--402},
title = {{Protection and the control of information sharing in Multics}},
url = {http://portal.acm.org/citation.cfm?doid=361011.361067},
volume = {17},
year = {1974}
}
@inproceedings{Levin1975,
address = {New York, New York, USA},
author = {Levin, R. and Cohen, E. and Corwin, W. and Pollack, F. and Wulf, W.},
title = {{Policy/mechanism separation in Hydra}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
doi = {10.1145/800213.806531},
pages = {132--140},
url = {http://portal.acm.org/citation.cfm?doid=800213.806531},
year = {1975}
}
@article{Dennis1966,
author = {Dennis, J. B. and van Horn, E. C.},
journal = {Communications of the ACM},
number = {3},
pages = {143--155},
title = {{Programming semantics for multiprogrammed computations}},
volume = {9},
year = {1966}
}
@article{Wulf1974,
author = {Wulf, W. and Cohen, E. and Corwin, W. and Jones, A. and Levin, R. and Pierson, C. and Pollack, F.},
doi = {10.1145/355616.364017},
journal = {Communications of the ACM},
number = {6},
pages = {337--345},
title = {{HYDRA: the kernel of a multiprocessor operating system}},
url = {http://portal.acm.org/citation.cfm?doid=355616.364017},
volume = {17},
year = {1974}
}
@article{Redell1980,
author = {Redell, D. D. and Dalal, Y. K. and Horsley, T. R. and Lauer, H. C. and Lynch, W. C. and McJones, P. R. and Murray, H. G. and Purcell, S. C.},
doi = {10.1145/358818.358822},
journal = {Communications of the ACM},
number = {2},
pages = {81--92},
title = {{Pilot: an operating system for a personal computer}},
url = {http://portal.acm.org/citation.cfm?doid=358818.358822},
volume = {23},
year = {1980}
}
@article{Saltzer1984,
author = {Saltzer, J. H. and Reed, D. P. and Clark, D. D.},
journal = {ACM Trans. Computer Systems (TOCS)},
pages = {509--512},
title = {{End-to-end arguments in system design}},
year = {1984}
}
@article{Pike1995,
author = {Pike, R. and Presotto, D. and Dorward, S. and Flandrena, B. and Thompson, K. and Trickey, H. and Winterbottom, P.},
journal = {Computing Systems},
number = {3},
pages = {221--254},
title = {{Plan 9 from Bell Labs}},
volume = {8},
year = {1995}
}
@article{Cao1996,
author = {Cao, P. and Felten, E. W. and Karlin, A. R. and Li, K.},
doi = {10.1145/235543.235544},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {4},
pages = {311--343},
title = {{Implementation and performance of integrated application-controlled file caching, prefetching, and disk scheduling}},
url = {http://portal.acm.org/citation.cfm?doid=235543.235544},
volume = {14},
year = {1996}
}
@inproceedings{Engler1995,
author = {Engler, D. R. and Kaashoek, M. F. and O'Toole, J. Jr.},
title = {{Exokernel: An operating system architecture for application-level resource management}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
url = {http://dl.acm.org/citation.cfm?id=224076},
year = {1995}
}
@article{Daley1968,
author = {Daley, R. C. and Dennis, J. B.},
journal = {Communications of the ACM},
number = {5},
pages = {306--312},
title = {{Virtual memory, processes, and sharing in MULTICS}},
volume = {11},
year = {1968}
}
@inproceedings{Kaashoek1997,
author = {Kaashoek, M. F. and Engler, D. R. and Ganger, G. R. and Brice讠M. and Hunt, R. and Mazi전. and Pinckney, T. and Grimm, R. and Jannotti, J. and Mackenzie, K.},
title = {{Application performance and flexibility on exokernel systems}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
url = {http://dl.acm.org/citation.cfm?id=266644},
year = {1997},
pages = {52--65}
}
@article{Hansen1970,
author = {Hansen, P. B.},
journal = {Communications of the ACM},
number = {4},
title = {{The nucleus of a multiprogramming system}},
volume = {13},
year = {1970},
pages = {238 --241}
}
@inproceedings{Clark1985,
author = {Clark, D. D.},
title = {{The structuring of systems using upcalls}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {171--180},
url = {http://dl.acm.org/citation.cfm?id=323645},
year = {1985}
}
@article{Lauer1979,
author = {Lauer, H. C. and Needham, R. M.},
doi = {10.1145/850657.850658},
journal = {ACM SIGOPS Operating Systems Review},
number = {2},
pages = {3--19},
title = {{On the duality of operating system structures}},
url = {http://portal.acm.org/citation.cfm?doid=850657.850658},
volume = {13},
year = {1979}
}
}}}
{{{
@inproceedings{Kaashoek1997,
author = {Kaashoek, M. F. and Engler, D. R. and Ganger, G. R. and Brice讠M. and Hunt, R. and Mazi전. and Pinckney, T. and Grimm, R. and Jannotti, J. and Mackenzie, K.},
title = {{Application performance and flexibility on exokernel systems}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
url = {http://dl.acm.org/citation.cfm?id=266644},
year = {1997},
pages = {52--65}
}
@inproceedings{Engler1995,
author = {Engler, D. R. and Kaashoek, M. F. and O'Toole, J. Jr.},
title = {{Exokernel: An operating system architecture for application-level resource management}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
url = {http://dl.acm.org/citation.cfm?id=224076},
year = {1995}
}
@inproceedings{Barham2003,
author = {Barham, P. and Dragovic, B. and Fraser, K. and Hand, S. and Harris, T. and Ho, A. and Neugebauer, R. and Pratt, I. and Warfield, A.},
title = {{Xen and the art of virtualization}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {164--177},
year = {2003}
}
@techreport{Tang1994,
author = {Tang, D. L. and Seltzer, M.},
institution = {TR-34-94, Harvard University},
title = {{Lies, damned lies, and file system benchmarks}},
year = {1994}
}
@techreport{Tang1995,
author = {Tang, D. L.},
institution = {TR-12-95, Harvard University},
title = {{Benchmarking filesystems}},
year = {1995}
}
@inproceedings{Ruwart2001,
author = {Ruwart, T.},
title = {{File system benchmarks, then, now, and tomorrow}},
booktitle = {Proc. IEEE Symp. Mass Storage Systems and Technologies},
doi = {10.1109/MSS.2001.10012},
pages = {117--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4022188},
year = {2001}
}
@inproceedings{Ellard2003,
author = {Ellard, D. and Seltzer, M.},
title = {{New NFS tracing tools and techniques for system analysis}},
booktitle = {Proc. Annual USENIX Conf. Large Installation Systems Administration},
pages = {73--86},
year = {2003}
}
@techreport{Small1997,
author = {Small, C. and Ghosh, N. and Saleeb, H. and Seltzer, M. and Smith, K.},
institution = {TR-16-97, Harvard University.},
title = {{Does systems research measure up?}},
year = {1997}
}
@inproceedings{Wood2008,
author = {Wood, T. and Cherkasova, L. and Ozonat, K.},
title = {{Profiling and modeling resource usage of virtualized applications}},
booktitle = {Proc. ACM/IFIP/USENIX Intl. Conf. on Middleware},
pages = {366--387},
year = {2008}
}
@inproceedings{Bershad1992,
author = {Bershad, B. K. and Draves, R. P. and Forin, A.},
title = {{Using microbenchmarks to evaluate system performance}},
booktitle = {Proc. IEEE WS. Workstation Operating Systems},
doi = {10.1109/WWOS.1992.275671},
pages = {148--153},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=275671},
year = {1992}
}
@article{Anderson1997,
author = {Anderson, J. M. and Weihl, W. E. and Berc, L. M. and Dean, J. and Ghemawat, S. and Henzinger, M. R. and Leung, S. T. A. and Sites, R. L. and Vandevoorde, M. T. and Waldspurger, C. A.},
doi = {10.1145/269005.266637},
journal = {ACM SIGOPS Operating Systems Review},
number = {5},
pages = {1--14},
title = {{Continuous profiling: Where hae all the cycles gone?}},
url = {http://portal.acm.org/citation.cfm?doid=269005.266637},
volume = {31},
year = {1997}
}
@article{Zhang1997,
author = {Zhang, X. and Wang, Z. and Gloy, N. and Chen, J. B. and Smith, M. D.},
doi = {10.1145/269005.266640},
journal = {ACM SIGOPS Operating Systems Review},
number = {5},
pages = {15--26},
title = {{System support for automatic profiling and optimization}},
url = {http://portal.acm.org/citation.cfm?doid=269005.266640},
volume = {31},
year = {1997}
}
@techreport{Smith2001,
author = {Smith, K. A.},
institution = {Harvard University},
title = {{Workload-specific file system benchmarks}},
url = {http://www.eecs.harvard.edu/~vino/fs-perf/papers/keith\_a\_smith\_thesis.pdf},
year = {2001}
}
@techreport{Xu2012,
author = {Xu, W.},
institution = {University of British Columbia},
title = {{IQ: A Whole-System Instrumentation Framework for Post Analysis}},
year = {2012}
}
@inproceedings{Liedtke1998,
author = {Liedtke, J. and Islam, N. and Jaeger, T. and Panteleenko, V. and Park, Yoonho},
title = {{Irreproducible benchmarks might be sometimes helpful}},
booktitle = {Proc. ACM SIGOPS European WS. Support for Composing Distributed Applications},
doi = {10.1145/319195.319232},
pages = {242--246},
url = {http://portal.acm.org/citation.cfm?doid=319195.319232},
year = {1998}
}
@inproceedings{Joukov2006,
author = {Joukov, N. and Traeger, A. and Iyer, R.},
title = {{Operating system profiling via latency analysis}},
booktitle = {Proc. Symp. Operating Systems Design and Implementation (OSDI)},
pages = {89--102},
url = {http://dl.acm.org/citation.cfm?id=1298465},
year = {2006}
}
@inproceedings{Yin2011,
author = {Yin, Z. and Ma, X. and Zheng, J. and Zhou, Y. and Bairavasundaram, L. N. and Pasupathy, S.},
title = {{An empirical study on configuration errors in commercial and open source systems}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
doi = {10.1145/2043556.2043572},
pages = {159},
url = {http://dl.acm.org/citation.cfm?doid=2043556.2043572},
year = {2011}
}
@article{Traeger2008,
author = {Traeger, A. and Zadok, E. and Joukov, N. and Wright, C. P.},
doi = {10.1145/1367829.1367831},
journal = {ACM Trans. Storage},
number = {2},
pages = {1--56},
title = {{A nine year study of file system and storage benchmarking}},
url = {http://portal.acm.org/citation.cfm?doid=1367829.1367831},
volume = {4},
year = {2008}
}
@inproceedings{Mogul1999,
author = {Mogul, J. C.},
title = {{Brittle metrics in operating systems research}},
booktitle = {Proc. IEEE WS. Hot Topics in Operating Systems},
doi = {10.1109/HOTOS.1999.798383},
pages = {90--95},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=798383},
year = {1999}
}
@article{Staelin2005,
author = {Staelin, C.},
doi = {10.1002/spe.665},
journal = {Software: Practice and Experience},
number = {11},
pages = {1079--1105},
title = {lmbench: an extensible micro-benchmark suite},
url = {http://doi.wiley.com/10.1002/spe.665},
volume = {35},
year = {2005}
}
@inproceedings{Urner1997,
author = {Urner, D. L.},
title = {{Pinpointing system performance issues}},
booktitle = {Proc. Systems Administration Conf. (LISA)},
pages = {141--153},
url = {https://www.usenix.org/legacyurl/papers-lisa-97-22},
year = {1997}
}
@inproceedings{Mcvoy1996,
author = {Mcvoy, L. and Staelin, C.},
title = {{lmbench: Portable tools for performance analysis}},
booktitle = {Proc. USENIX Annual Technical Conf.},
number = {January},
pages = {279--294},
year = {1996}
}
@article{Brown1997,
author = {Brown, A. B. and Seltzer, M. I.},
doi = {10.1145/258623.258690},
title = {{Operating system benchmarking in the wake of lmbench}},
journal = {ACM SIGMETRICS Performance Evaluation Review},
month = jun,
number = {1},
pages = {214--224},
url = {http://portal.acm.org/citation.cfm?doid=258623.258690},
volume = {25},
year = {1997}
}
@inproceedings{Mogul1992,
author = {Mogul, J. C.},
title = {{SPECmarks are leading us astray}},
booktitle = {Proc. WS. Workstation Operating Systems},
doi = {10.1109/WWOS.1992.275669},
pages = {160--161},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=275669},
year = {1992}
}
@inproceedings{Appel1991,
author = {Appel, A. W. and Li, K.},
title = {{Virtual memory primitives for user programs}},
booktitle = {Proc. Intl. Conf. Architectural Support for Programming Languages and Operating Systems},
doi = {10.1145/106973.106984},
number = {1},
pages = {213--216},
volume = {28},
year = {1991}
}
@techreport{Anderson1993,
author = {Anderson, T. E.},
institution = {University of California, Berkeley, Computer Science Division.},
pages = {1--3},
title = {{The case for application-specific operating systems}},
url = {http://fmg-www.cs.ucla.edu/classes/239\_2.fall98/papers/app\_specific.ps},
year = {1993}
}
@inproceedings{Ousterhout1990,
author = {Ousterhout, J. K.},
title = {{Why aren't operating systems getting faster as fast as hardware?}},
booktitle = {Proc. USENIX Summer Conf.},
pages = {1--10},
url = {http://ukpmc.ac.uk/abstract/CIT/234323},
year = {1990}
}
@inproceedings{Roselli2000,
author = {Roselli, D. and Lorch, J. R. and Anderson, T. E.},
title = {{A comparison of file system workloads}},
booktitle = {Proc. USENIX Annual Technical Conf.},
year = {2000}
}
@inproceedings{Clark2004,
author = {Clark, B. and Deshane, T. and Dow, E. and Evanchik, S. and Finlayson, M. and Herne, J.},
title = {{Xen and the art of repeated research}},
booktitle = {Proc. USENIX Annual Technical Conf.},
pages = {135--144},
year = {2004}
}
@inproceedings{Baker1991,
author = {Baker, G. and Hartmart, J. H. and Ousterhout, J. K. and Shirriff, K. W.},
title = {{Measurements of a distributed file system}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
pages = {198--212},
year = {1991}
}
@inproceedings{Vogels1999,
author = {Vogels, W.},
title = {{File system usage in Windows NT 4.0}},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
doi = {10.1145/319151.319158},
pages = {93--109},
url = {http://portal.acm.org/citation.cfm?doid=319151.319158},
year = {1999}
}
@inproceedings{Seltzer1999,
author = {Seltzer, M. and Krinsky, D. and Smith, K.},
title = {{The case for application-specific benchmarking}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=798385},
booktitle = {Proc. IEEE WS. Hot Topics in Operating Systems},
doi = {10.1109/HOTOS.1999.798385},
pages = {102--107},
year = {1999}
}
@inproceedings{Chou2001,
author = {Chou, A. and Yang, J. and Chelf, B. and Hallem, S. and Engler, D. R.},
doi = {10.1145/502039.502042},
pages = {73--88},
title = {{An empirical study of operating systems errors}},
url = {http://portal.acm.org/citation.cfm?doid=502034.502042},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
year = {2001}
}
@inproceedings{Koch1993,
author = {Koch, P.T.},
doi = {10.1109/IWOOOS.1993.324913},
pages = {162--166},
title = {{Performance evaluation of two software distributed shared memory models}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=324913},
booktitle = {Proc. Intl. WS. Object Orientation in Operating Systems},
year = {1993}
}
@article{Beaudry1996,
author = {Beaudry, M. D.},
doi = {10.1109/TC.1978.1675145},
journal = {IEEE Trans. Computers},
number = {6},
pages = {540--547},
title = {{Performance-related reliability measures for computing systems}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1675145},
volume = {C-27},
year = {1978}
}
@article{Heidelberger1984,
author = {Heidelberger, P.},
doi = {10.1109/TC.1984.1676394},
journal = {IEEE Trans. Computers},
number = {12},
pages = {1195--1220},
title = {{Computer performance evaluation methodology}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1676394},
volume = {C-33},
year = {1984}
}
@inproceedings{Endo1996a,
author = {Endo, Y. and Wang, Z. and Chen, J. B. and Seltzer, M.},
doi = {10.1145/238721.238775},
pages = {185--199},
title = {{Using latency to evaluate interactive system performance}},
url = {http://portal.acm.org/citation.cfm?doid=238721.238775},
booktitle = {Proc. USENIX Symp. Operating Systems Design and Implementation (OSDI)},
year = {1996}
}
@article{Miyazaki1982,
author = {Miyazaki, M. and Matsuzawa, S. and Obata, S. and Noguchi, S.},
doi = {10.1007/3-540-11604-4\_52},
journal = {Operating Systems Engineering: Lecture Notes in Computer Science},
pages = {143--158},
title = {{On pseudo wordload for performance measurement of interactive systems}},
url = {http://link.springer.com/chapter/10.1007/3-540-11604-4\_52?null},
volume = {143},
year = {1982}
}
@article{Chen1996,
author = {Chen, J. B. and Endo, Y. and Chan, K. and Mazi전. and Dias, A. and Seltzer, M. and Smith, M. D.},
doi = {10.1145/225535.225536},
journal = {ACM Transactions on Computer Systems},
number = {1},
pages = {3--40},
title = {{The measured performance of personal computer operating systems}},
url = {http://portal.acm.org/citation.cfm?doid=225535.225536},
volume = {14},
year = {1996}
}
@article{Thereska2006,
author = {Thereska, E. and Salmon, B. and Strunk, J. and Wachs, M. and Abd-El-Malek, M. and Lopez, J. and Ganger, G. R.},
doi = {10.1145/1140103.1140280},
journal = {ACM SIGMETRICS Performance Evaluation Review},
number = {1},
pages = {3},
title = {{Stardust: Tracking activity in a distributed storage system}},
url = {http://portal.acm.org/citation.cfm?doid=1140103.1140280},
volume = {34},
year = {2006}
}
@article{Calingaert1967,
author = {Calingaert, P.},
doi = {10.1145/363018.363040},
journal = {Communications of the ACM},
number = {1},
pages = {12--18},
title = {{System performance evaluation: survey and appraisal}},
url = {http://portal.acm.org/citation.cfm?doid=363018.363040},
volume = {10},
year = {1967}
}
@article{Ousterhout1985,
author = {Ousterhout, J. K. and {Da Costa}, H. and Harrison, D. and Kunze, J. A. and Kupfer, M. and Thompson, J. G.},
doi = {10.1145/323627.323631},
journal = {ACM SIGOPS Operating Systems Review},
number = {5},
pages = {15--24},
title = {{A trace-driven analysis of the UNIX 4.2 BSD file system}},
url = {http://portal.acm.org/citation.cfm?doid=323627.323631},
volume = {19},
year = {1985}
}
@article{Lehmann1980,
author = {Lehmann, A.},
doi = {10.1145/1009375.806149},
journal = {ACM SIGMETRICS Performance Evaluation Review},
number = {2},
pages = {43--54},
title = {{Performance evaluation and prediction of storage hierarchies}},
url = {http://portal.acm.org/citation.cfm?doid=1009375.806149},
volume = {9},
year = {1980}
}
@article{Park1990,
author = {Park, A. and Becker, J. C. and Lipton, R. J.},
doi = {10.1145/88237.88242},
journal = {ACM SIGARCH Computer Architecture News},
number = {2},
pages = {45--52},
title = {{IOStone: a synthetic file system benchmark}},
url = {http://portal.acm.org/citation.cfm?doid=88237.88242},
volume = {18},
year = {1990}
}
@article{Howard1988,
author = {Howard, J. H. and Kazar, M. L. and Menees, S. G. and Nichols, D. A. and Satyanarayanan, M. and Sidebotham, R. N. and West, M. J.},
doi = {10.1145/35037.35059},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {1},
pages = {51--81},
title = {{Scale and performance in a distributed file system}},
url = {http://portal.acm.org/citation.cfm?doid=35037.35059},
volume = {6},
year = {1988}
}
@article{Douceur1981,
author = {Douceur, J. R. and Bolosky, W. J.},
doi = {10.1145/301464.301480},
title = {{A large-scale study of file-system contents}},
url = {http://portal.acm.org/citation.cfm?doid=301464.301480},
journal = {ACM SIGMETRICS Performance Evaluation Review},
volume = {27},
number = {1},
pages = {59--70},
year = {1981}
}
@article{Hartig1997,
author = {H笠H. and Hohmuth, M. and Liedtke, J. and Sch⧬ S.},
doi = {10.1145/269005.266660},
journal = {ACM SIGOPS Operating Systems Review},
number = {5},
pages = {66--77},
title = {{The performance of ⮥l-based systems}},
url = {http://portal.acm.org/citation.cfm?doid=269005.266660},
volume = {31},
year = {1997}
}
@article{Lucas1971,
author = {Lucas, H.},
doi = {10.1145/356589.356590},
journal = {ACM Computing Surveys},
number = {3},
pages = {79--91},
title = {{Performance evaluation and monitoring}},
url = {http://portal.acm.org/citation.cfm?doid=356589.356590},
volume = {3},
year = {1971}
}
@inproceedings{Satyanarayanan1981,
author = {Satyanarayanan, M.},
doi = {10.1145/800216.806597},
pages = {96--108},
title = {{A study of file sizes and functional lifetimes}},
url = {http://portal.acm.org/citation.cfm?doid=800216.806597},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
year = {1981}
}
@inproceedings{Chen1993,
author = {Chen, J. B. and Bershad, B. N.},
doi = {10.1145/168619.168629},
pages = {120--133},
title = {{The impact of operating system structure on memory system performance}},
url = {http://portal.acm.org/citation.cfm?doid=168619.168629},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
year = {1993}
}
@article{Sreenivasan1974,
author = {Sreenivasan, K. and Kleinman, A. J.},
doi = {10.1145/360860.360863},
journal = {Communications of the ACM},
number = {3},
pages = {127--133},
title = {{On the construction of a representative synthetic workload}},
url = {http://portal.acm.org/citation.cfm?doid=360860.360863},
volume = {17},
year = {1974}
}
@inproceedings{Noble1994,
author = {Noble, B. D. and Satyanarayanan, M.},
doi = {10.1145/183018.183033},
pages = {138--149},
title = {{An empirical study of a highly available file system}},
url = {http://portal.acm.org/citation.cfm?doid=183018.183033},
booktitle = {Proc. ACM SIGMETRICS Intl. Conf. Measurement and Modeling of Computer Systems},
year = {1994}
}
@inproceedings{Karush1969,
author = {Karush, A. D.},
doi = {10.1145/961053.961103},
pages = {159},
title = {{Two approaches for measuring the performance of time-sharing systems}},
url = {http://portal.acm.org/citation.cfm?doid=961053.961103},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
year = {1969}
}
@article{Spasojevic1996a,
author = {Spasojevic, M. and Satyanarayanan, M.},
doi = {10.1145/227695.227698},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {2},
pages = {200--222},
title = {{An empirical study of a wide-area distributed file system}},
url = {http://portal.acm.org/citation.cfm?doid=227695.227698},
volume = {14},
year = {1996}
}
@inproceedings{Cantrell1968,
author = {Cantrell, H. N. and Ellison, A. L.},
doi = {10.1145/1468075.1468108},
pages = {213},
title = {{Multiprogramming system performance measurement and analysis}},
url = {http://portal.acm.org/citation.cfm?doid=1468075.1468108},
booktitle = {Proc. Spring Joint Computer Conf. (AFIPS)},
year = {1968}
}
@article{Lazowska1986a,
author = {Lazowska, E. D. and Zahorjan, J. and Cheriton, D. R. and Zwaenepoel, W.},
doi = {10.1145/6420.6423},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {3},
pages = {238--268},
title = {{File access performance of diskless workstations}},
url = {http://portal.acm.org/citation.cfm?doid=6420.6423},
volume = {4},
year = {1986}
}
@inproceedings{Dujomovic1979,
author = {Dujomovic, J. J.},
doi = {10.1145/1013608.805463},
number = {1},
pages = {259--267},
title = {{Criteria for computer performance analysis}},
url = {http://portal.acm.org/citation.cfm?doid=1013608.805463},
volume = {11},
booktitle = {Proc. ACM SIGMETRICS Intl. Conf. Measurement and Modeling of Computer Systems},
year = {1979}
}
@article{Chen1994,
author = {Chen, P. M. and Patterson, D. A.},
doi = {10.1145/195792.195812},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {4},
pages = {308--339},
title = {{A new approach to I/O performance evaluation: Self-scaling I/O benchmarks, predicted I/O performance}},
url = {http://portal.acm.org/citation.cfm?doid=195792.195812},
volume = {12},
year = {1994}
}
@article{Saavedra1996,
author = {Saavedra, R. H. and Smith, A. J.},
doi = {10.1145/235543.235545},
journal = {ACM Trans. Computer Systems (TOCS)},
number = {4},
pages = {344--384},
title = {{Analysis of benchmark characteristics and benchmark performance prediction}},
url = {http://portal.acm.org/citation.cfm?doid=235543.235545},
volume = {14},
year = {1996}
}
@inproceedings{Oliver1974,
author = {Oliver, P. and Baird, G. and Cook, M. and Johnson, A. and Hoyt, P.},
doi = {10.1145/1500175.1500265},
pages = {431},
title = {{An experiment in the use of synthetic programs for system benchmarking}},
url = {http://portal.acm.org/citation.cfm?doid=1500175.1500265},
booktitle = {Proc. Spring Joint Computer Conf. (AFIPS)},
year = {1974}
}
@article{Kimbleton1972,
author = {Kimbleton, S. R.},
doi = {10.1145/361454.361480},
journal = {Communications of the ACM},
number = {7},
pages = {586--590},
title = {{The role of computer system models in performance evaluation}},
url = {http://portal.acm.org/citation.cfm?doid=361454.361480},
volume = {15},
year = {1972}
}
@inproceedings{Campbell1968,
author = {Campbell, D. J. and Heffner, W. J.},
doi = {10.1145/1476589.1476704},
pages = {903},
title = {{Measurement and analysis of large operating systems during system development}},
url = {http://portal.acm.org/citation.cfm?doid=1476589.1476704},
booktitle = {Proc. Fall Joint Computer Conf. (AFIPS)},
year = {1968}
}
@inproceedings{Transarc1994,
author = {Transarc, M. S. and Spasojevic, M.},
doi = {10.1.1.46.8312},
pages = {1--17},
title = {{A usage profile and evaluation of a wide-area distributed file system}},
url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.46.8312},
booktitle = {Proc. USENIX Winter Technical Conf.},
year = {1994}
}
@inproceedings{Rosenblum1995,
author = {Rosenblum, M. and Bugnion, E. and Herrod, S. A. and Witchel, E. and Gupta, A.},
doi = {10.1145/224056.224078},
pages = {285--298},
title = {{The impact of architectural trends on operating system performance}},
url = {http://portal.acm.org/citation.cfm?doid=224056.224078},
booktitle = {Proc. ACM Symp. Operating Systems Principles (SOSP)},
year = {1995}
}
@article{Perl1996,
author = {Perl, S. E. and Sites, R. L.},
doi = {10.1145/248155.238773},
journal = {ACM SIGOPS Operating Systems Review},
number = {SI},
pages = {169--183},
title = {{Studies of Windows NT performance using dynamic execution traces}},
url = {http://portal.acm.org/citation.cfm?doid=248155.238773},
volume = {30},
year = {1996}
}
@article{Gupta1991,
author = {Gupta, A. and Tucker, A. and Urushibara, S.},
doi = {10.1145/107972.107985},
journal = {ACM SIGMETRICS Performance Evaluation Review},
number = {1},
pages = {120--132},
title = {{The impact of operating system scheduling policies and synchronization methods of performance of parallel applications}},
url = {http://portal.acm.org/citation.cfm?doid=107972.107985},
volume = {19},
year = {1991}
}
@article{Cao1995,
author = {Cao, P. and Felten, E. W. and Karlin, A. R. and Li, K.},
doi = {10.1145/223586.223608},
journal = {ACM SIGMETRICS Performance Evaluation Review},
number = {1},
pages = {188--197},
title = {{A study of integrated prefetching and caching strategies}},
url = {http://portal.acm.org/citation.cfm?doid=223586.223608},
volume = {23},
year = {1995}
}
}}}
{{{
@inproceedings{Amar2004,
author = {Amar, R. and Eagan, J. and Stasko, J. T.},
title = {{Low-level components of analytic activity in information visualization}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2005.1532136},
pages = {111--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532136},
year = {2004}
}
@incollection{Card1999,
author = {Card, S. K. and Mackinlay, J. D. and Shneiderman, B.},
booktitle = {Readings in Information Visualization: Using Vision to Think},
chapter = {1},
pages = {1--34},
publisher = {Morgan Kaufmann},
title = {{Information visualization}},
year = {1999}
}
@inproceedings{Appert2006,
author = {Appert, C. and Fekete, J. D.},
title = {{OrthoZoom scroller: 1D multi-scale navigation}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {30},
publisher = {ACM},
url = {http://portal.acm.org/citation.cfm?id=1124776},
volume = {06pages},
year = {2006}
}
@inproceedings{Auber2003,
author = {Auber, D. and Chiricota, Y. and Jourdan, F. and MelanǮ},
title = {{Multiscale visualization of small world networks}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2003.1249011},
pages = {75--81},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1249011},
year = {2003}
}
@article{Becker1996a,
author = {Becker, R. A and Cleveland, W. S. and Shyu, M. J.},
doi = {10.2307/1390777},
journal = {Computational and Graphical Statistics},
number = {2},
pages = {123},
title = {{The Visual Design and Control of Trellis Display}},
url = {http://www.jstor.org/stable/1390777?origin=crossref},
volume = {5},
year = {1996}
}
@article{Cleveland1984a,
author = {Cleveland, W. S. and ~McGill, R.},
journal = {J. American Statistical Association},
number = {387},
pages = {531-- 554},
title = {{Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods}},
url = {http://www.jstor.org/stable/2288400},
volume = {79},
year = {1984}
}
@article{Cockburn2008a,
author = {Cockburn, A. and Karlson, A.},
doi = {10.1145/1456650.145662},
journal = {ACM Computing Surveys},
pages = {1--42},
title = {{A review of overview+ detail, zooming, and focus+ context interfaces}},
url = {http://portal.acm.org/citation.cfm?id=1456652\&dl=ACM},
volume = {41},
year = {2008}
}
@inproceedings{Davidson2001,
author = {Davidson, G. S. and Wylie, B. N. and Boyack, K. W.},
title = {{Cluster stability and the use of noise in interpretation of clustering}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2001.963275},
pages = {23--30},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=963275},
year = {2001}
}
@article{Dwyer2006,
author = {Dwyer, T. and Koren, Y. and Marriott, K.},
doi = {10.1109/TVCG.2006.156},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {821--8},
title = {{IPSEP-COLA: an incremental procedure for separation constraint layout of graphs}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080805},
volume = {12},
year = {2006}
}
@inproceedings{Fua1999,
author = {Fua, Y. H. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Hierarchical parallel coordinates for exploration of large datasets}},
booktitle = {Proc. IEEE Visualization},
pages = {43--50},
url = {http://doi.ieeecomputersociety.org/10.1109/VISUAL.1999.809866},
volume = {99},
year = {1999}
}
@article{Gansner2004,
author = {Gansner, E. R. and Koren, Y. and North, S. C.},
doi = {10.1109/TVCG.2005.66},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {457--68},
title = {{Topological fisheye views for visualizing large graphs}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16138555},
volume = {11},
year = {2004}
}
@article{Heer2006,
author = {Heer, J. and Agrawala, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {2},
title = {{Multi-Scale Banking to 45 Degrees}},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.95.3461\&rep=rep1\&type=pdf},
volume = {12},
year = {2006}
}
@article{Heer2007,
author = {Heer, J. and Robertson, G. G.},
doi = {10.1109/TVCG.2007.70539},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1240--1247},
title = {{Animated transitions in statistical data graphics}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968070},
volume = {13},
year = {2007}
}
@article{Henry2006,
author = {Henry, N. and Fekete, J. D.},
doi = {10.1109/TVCG.2006.160},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {677--84},
title = {{MatrixExplorer: a dual-representation system to explore social networks.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080787},
volume = {12},
year = {2006}
}
@article{Herman2000,
author = {Herman, I. and MelanǮ and Marshall, M. S.},
doi = {10.1109/2945.841119},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {24--43},
title = {{Graph visualization and navigation in information visualization: A survey}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=841119},
volume = {6},
year = {2000}
}
@article{Holten2006,
author = {Holten, D.},
doi = {10.1109/TVCG.2006.147},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {741--8},
title = {{Hierarchical edge bundles: visualization of adjacency relations in hierarchical data}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080795},
volume = {12},
year = {2006}
}
@article{Hornb෬
author = {Hornb뮠and Hertzum, M.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {2},
pages = {6},
publisher = {ACM},
title = {{Untangling the usability of fisheye menus}},
url = {http://portal.acm.org/citation.cfm?id=1275511.1275512},
volume = {14},
year = {2007}
}
@inproceedings{Huber2005,
author = {Huber, D. E. and Healey, C. G.},
title = {{Visualizing Data with Motion}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/VISUAL.2005.1532838},
pages = {527--534},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532838},
year = {2005}
}
@article{Ingram2009,
author = {Ingram, S. and Munzner, T. and Olano, M.},
doi = {10.1109/TVCG.2008.85},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {249--61},
title = {{Glimmer: multilevel MDS on the GPU}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19147889},
volume = {15},
year = {2009}
}
@article{Keim1994,
author = {Keim, D. A. and Kriegel, H. P.},
doi = {10.1109/38.310723},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {5},
pages = {40--49},
title = {{VisDB: database exploration using multidimensional visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=310723},
volume = {14},
year = {1994}
}
@inproceedings{Kindlmann2002,
author = {Kindlmann, G. and Reinhard, E. and Creem, S.},
title = {{Face-based luminance matching for perceptual colormap generation}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {299--306},
url = {http://portal.acm.org/citation.cfm?id=602099.602145\&type=series},
year = {2002}
}
@inproceedings{Lamping1995,
author = {Lamping, J. and Rao, R. and Pirolli, P.},
title = {{A focus+ context technique based on hyperbolic geometry for visualizing large hierarchies}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {408},
publisher = {ACM},
url = {http://portal.acm.org/citation.cfm?id=223956},
year = {1995}
}
@article{MacEachren2000,
author = {MacEachren, A. M. and Xiping, D. and Hardisty, F. and Lengerich, G.},
doi = {10.1109/INFVIS.2003.1249006},
journal = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {31--38},
title = {{Exploring high-D spaces with multiform matrices and small multiples}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1249006},
year = {2000}
}
@article{Meyer2009,
author = {Meyer, M. and Munzner, T. and Pfister, H.},
doi = {10.1109/TVCG.2009.167},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {897--904},
title = {{MizBee: a multiscale synteny browser}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834152},
volume = {15},
year = {2009}
}
@incollection{Munzner2009a,
author = {Munzner, T.},
title = {{Visualization}},
booktitle = {Fundamentals of Graphics},
chapter = {27},
edition = {3rd},
editor = {Shirley, P. and Marschner, S.},
pages = {675--707},
publisher = {AK Peters},
year = {2009}
}
@article{Munzner2008,
author = {Munzner, T.},
journal = {Lecture Notes in Computer Science},
pages = {134--153},
publisher = {Springer},
title = {{Process and pitfalls in writing information visualization research papers}},
url = {http://www.springerlink.com/index/98p61l258kn5780l.pdf},
volume = {4950},
year = {2008}
}
@article{Munzner2009,
author = {Munzner, T.},
doi = {10.1109/TVCG.2009.111},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {921--928},
title = {{A nested model for visualization design and validation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834155},
volume = {15},
year = {2009}
}
@article{Munzner2003,
author = {Munzner, T. and GuimbretiƮ and Tasiran, S. and Zhang, L. and Zhou, Y.},
journal = {ACM Trans. Graphics (TOG)},
number = {3},
pages = {462},
publisher = {ACM},
title = {{TreeJuxtaposer: scalable tree comparison using Focus+ Context with guaranteed visibility}},
url = {http://portal.acm.org/citation.cfm?id=882262.882291},
volume = {22},
year = {2003}
}
@inproceedings{Plaisant1998,
author = {Plaisant, C. and Grosjean, J. and Bederson, B. B.},
title = {{SpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2002.1173148},
pages = {57--64},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173148},
year = {1998}
}
@article{Robertson,
author = {Robertson, G. G. and Fernandez, R. and Fisher, D. and Lee, B. and Stasko, J. T.},
title = {{Effectiveness of animation in trend visualization}},
doi = {10.1109/TVCG.2008.125},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1325--32},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988980},
volume = {14},
year = {2008}
}
@article{Slingsby2009,
author = {Slingsby, A. and Dykes, J. and Wood, J.},
doi = {10.1109/TVCG.2009.128},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {977--84},
title = {{Configuring hierarchical layouts to address research questions}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834162},
volume = {15},
year = {2009}
}
@article{Stolte2002,
author = {Stolte, C. and Tang, D. and Hanrahan, P.},
doi = {10.1109/2945.981851},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {52--65},
title = {{Polaris: a system for query, analysis, and visualization of multidimensional relational databases}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=981851},
volume = {8},
year = {2002}
}
@article{Stone2005,
author = {Stone, M. C.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {4},
pages = {78--85},
title = {{Representing colors as three numbers}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16060577},
volume = {25},
year = {2005}
}
@article{Trafton2000,
author = {Trafton, J. G. and Kirschenbaum, S. S. and Tsui, T. L. and Miyamoto, R. T. and Ballas, J. A. and Raymond, P. D.},
doi = {10.1006/ijhc.2000.0419},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {827--850},
title = {{Turning pictures into numbers: extracting and generating information from complex visualizations}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904191},
volume = {53},
year = {2000}
}
@inproceedings{VanWijk1999,
author = {Wijk, J. J. van and Selow, E. R. van},
title = {{Cluster and calendar based visualization of time series data}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {4--9},
year = {1999}
}
@inproceedings{Ward,
author = {Ward, M. O. and Rundensteiner, E. A.},
title = {{Interactive hierarchical dimension ordering, spacing and filtering for exploration of high dimensional datasets}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2003.1249015},
pages = {105--112},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1249015},
year = {2003}
}
@inproceedings{Weaver2004,
author = {Weaver, C.},
title = {{Building Highly-Coordinated Visualizations in Improvise}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2004.12},
pages = {159--166},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382904},
year = {2004}
}
@article{Weaver2007,
author = {Weaver, C. and Fyfe, D. and Robinson, A. and Holdsworth, D. and Peuquet, D. and ~MacEachren, A. M.},
doi = {10.1057/palgrave.ivs.9500145},
journal = {Information Visualization},
pages = {89--103},
title = {{Visual exploration and analysis of historic hotel visits}},
url = {http://www.palgrave-journals.com/doifinder/10.1057/palgrave.ivs.9500145},
year = {2007}
}
@inproceedings{Wilkinson2005,
author = {Wilkinson, L. and Anand, A. and Grossman, R.},
title = {{Graph-theoretic scagnostics}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.62.6148\&rep=rep1\&type=pdf},
year = {2005}
}
@article{Willett2007,
author = {Willett, W. and Heer, J. and Agrawala, M.},
doi = {10.1109/TVCG.2007.70589},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1129--1136},
title = {{Scented widgets: improving navigation cues with embedded visualizations}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968056},
volume = {13},
year = {2007}
}
@article{Yannier2008,
author = {Yannier, N. and Basdogan, C. and Tasiran, S. and Sen, O. L.},
doi = {10.1109/TOH.2008.16},
journal = {IEEE Trans. Haptics},
number = {2},
pages = {130--141},
title = {{Using Haptics to Convey Cause-and-Effect Relations in Climate Visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4657354},
volume = {1},
year = {2008}
}
@article{Yost2006,
author = {Yost, B. and North, C.},
doi = {10.1109/TVCG.2006.184},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {837--844},
title = {{The perceptual scalability of visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080807},
volume = {12},
year = {2006}
}
}}}
{{{
@inproceedings{SantosAmorim2012,
author = {{Santos Amorim}, E. P. dos and Brazil, E. V. and Daniels, J. and Joia, P. and Nonato, L. G. and Sousa, M. C.},
title = {{iLAMP: Exploring high-dimensional spacing through backward multidimensional projection}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {53--62},
year = {2012}
}
@article{Sips2009,
author = {Sips, M. and Neubert, B. and Lewis, J. P. and Hanrahan, P.},
title = {{Selecting good views of high-dimensional data using class consistency}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {831--838},
volume = {28},
year = {2009}
}
@inproceedings{Lewis2012,
author = {Lewis, J. M. and van der Maaten, L. and de Sa, V. R.},
title = {{A behavioral investigation of dimensionality reduction}},
booktitle = {Proc. Cognitive Science Society},
pages = {671--676},
year = {2012}
}
@article{Bergner2013,
author = {Bergner, S. and Sedlmair, M. and M\"{o}ller, T. and Abdolyousefi, S. N. and Saad, A.},
title = {{ParaGlide: interactive parameter space partitioning for computer simulations}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {9},
pages = {1499--1512},
volume = {19},
year = {2013}
}
@article{Dasgupta2010,
author = {Dasgupta, A. and Kosara, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1017--1026},
title = {{Pargnostics: screen-space metrics for parallel coordinates.}},
volume = {16},
year = {2010}
}
@article{Guo2003a,
author = {Guo, D.},
journal = {Information Visualization},
number = {4},
pages = {232--246},
title = {{Coordinating computational and visual approaches for interactive feature selection and multivariate clustering}},
volume = {2},
year = {2003}
}
@article{Cui2006,
author = {Cui, Q. and Ward, M. O. and Rundensteiner, E. A. and Yang, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {709--716},
title = {{Measuring data abstraction quality in multiresolution visualizations}},
volume = {12},
year = {2006}
}
@inproceedings{Schneidewind2006,
author = {Schneidewind, J. and Sips, M. and Keim, D.},
title = {{Pixnostics: Towards measuring the value of visualization}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {199--206},
year = {2006}
}
@inproceedings{Ferdosi2010,
author = {Ferdosi, B. J. and Buddelmeijer, H. and Trager, S. and Wilkinson, M. H. F. and Roerdink, J. B. T. M.},
title = {{Finding and visualizing relevant subspaces for clustering high-dimensional astronomical data using connected morphological operators}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {35--42},
year = {2010}
}
@article{Holten2010,
author = {Holten, D. and {Van Wijk}, J. J.},
journal = {Computer Graphics Forum},
number = {3},
pages = {793--802},
title = {{Evaluation of cluster identification performance for different PCP variants}},
volume = {29},
year = {2010}
}
@inproceedings{Tatu2009,
author = {Tatu, A. and Albuquerque, G. and Eisemann, M. and Schneidewind, J. and Thiesel, H. and Magnor, M. and Keim, D. A.},
title = {{Combining automated analysis and visualization techniques for effective exploration of high-dimensional data}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {59--66},
year = {2009}
}
@article{Sedlmair2013,
author = {Sedlmair, M. and Munzner, T. and Tory, M.},
number = {12},
pages = {2634--2643},
title = {{Empirical guidance on scatterplot and dimension reduction technique choices}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
volume = {19},
year = {2013}
}
@article{Rosario2004,
author = {Rosario, G. E. and Rundensteiner, E. A. and Brown, D. C. and Ward, M. O. and Huang, S.},
journal = {Information Visualization},
number = {2},
pages = {80--95},
title = {{Mapping nominal values to numbers for effective visualization}},
volume = {3},
year = {2004}
}
@article{Chen2001,
author = {Chen, J. X. and Wang, S.},
journal = {Computer Sci. Eng.},
number = {5},
pages = {110--112},
title = {{Data visualization: Parallel coordinates and dimension reduction}},
volume = {3},
year = {2001}
}
@inproceedings{Pobitzer2012,
author = {Pobitzer, A. and Lez, A. and Matkovic, K. and Hauser, H.},
title = {{A statistics-based dimension reduction of the space of path line attributes for interactive visual flow analysis}},
booktitle = {Proc. IEEE Pacific Visualization Symp. (PacificVis)},
pages = {113--120},
year = {2012}
}
@article{Albuquerque2010a,
author = {Albuquerque, Georgia and Eisemann, Martin and Lehmann, Dirk J. and Theisel, Holger and Magnor, Marcus},
journal = {2010 IEEE Symposium on Visual Analytics Science and Technology},
pages = {19--26},
title = {{Improving the visual analysis of high-dimensional datasets using quality measures}},
year = {2010}
}
@article{Tatu2010a,
author = {Tatu, A. and Albuquerque, G. and Eisemann, M. and Bak, P. and Theisel, H. and Magnor, M. and Keim, D. A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {584--597},
title = {{Automated analytical methods to support visual exploration of high-dimensional data.}},
volume = {17},
year = {2010}
}
@inproceedings{Choi2010,
author = {Choi, J. Y. and Bae, S.-H. and Qiu, X. and Fox, G.},
title = {{High performance dimension reduction and visualization for large high-dimensional data analysis}},
booktitle = {Proc. IEEE/ACM Intl. Conf. Cluster, Cloud and Grid Computing (CCGrid)},
pages = {331--340},
year = {2010}
}
@inproceedings{Choo2009,
author = {Choo, J. and Bohn, S. and Park, H.},
title = {{Two-stage framework for visualization of clustered high dimensional data}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {67--74},
year = {2009}
}
@article{Oeltze2007a,
author = {Oeltze, S. and Doleisch, H. and Hauser, H. and Muigg, P. and Preim, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1392--13009},
title = {{Interactive visual analysis of perfusion data}},
volume = {13},
year = {2007}
}
@inproceedings{Yang2006,
author = {Yang, J. and Fan, J. and Hubball, D. and Gao, Y. and Luo, H. and Ribarsky, W. and Ward, M.},
title = {{Semantic image browser: Bridging information visualization with automated intelligent image analysis}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {191--198},
year = {2006}
}
@inproceedings{Artero2006,
author = {Artero, A. O. and de Oliveira, M. C. F. and Levkowitz, H.},
title = {{Enhanced high dimensional data visualization through dimension reduction and attribute arrangement}},
booktitle = {Proc. Intl. Conf. Information Visualization (IV)},
pages = {707--712},
year = {2006}
}
@article{Wu2005,
author = {Wu, S. and Chow, T. W. S.},
journal = {IEEE Trans. Neural Networks},
number = {6},
pages = {1362--80},
title = {{PRSOM: A new visualization method by hybridizing multidimensional scaling and self-organizing map}},
volume = {16},
year = {2005}
}
@article{Fernstad2012,
author = {Fernstad, S. J. and Shaw, J. and Johansson, J.},
doi = {10.1177/1473871612460526},
journal = {Information Visualization},
number = {1},
pages = {44--64},
title = {{Quality-based guidance for exploratory dimensionality reduction}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871612460526},
volume = {12},
year = {2012}
}
@article{Gerber2010,
author = {Gerber, S. and Bremer, P. T. and Pascucci, V. and Whitaker, R.},
doi = {10.1109/TVCG.2010.213},
file = {:Users/matthewbrehmer/Dropbox/Academics/Mendeley/Gerber et al/Gerber et al.\_2010.pdf:pdf},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1271--80},
title = {{Visual exploration of high dimensional scalar functions}},
url = {http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3099238\&tool=pmcentrez\&rendertype=abstract},
volume = {16},
year = {2010}
}
@inproceedings{Cui2008,
author = {Cui, Y. and Roto, V.},
title = {{How people use the web on mobile devices}},
booktitle = {Proc. Intl. Conf. World Wide Web (WWW)},
doi = {10.1145/1367497.1367619},
pages = {905--914},
url = {http://portal.acm.org/citation.cfm?doid=1367497.1367619},
year = {2008}
}
@techreport{Sedlmair2012,
author = {Sedlmair, M. and Brehmer, M. and Ingram, S. and Munzner, T.},
institution = {University of British Columbia},
title = {{Dimensionality Reduction in the Wild: Gaps and Guidance}},
year = {2012}
}
@article{Langley2011,
author = {Langley, P.},
doi = {10.1007/s10994-011-5242-y},
journal = {Machine Learning},
number = {3},
pages = {275--279},
title = {{The changing science of machine learning}},
url = {http://www.springerlink.com/index/10.1007/s10994-011-5242-y},
volume = {82},
year = {2011}
}
@article{VanIterson2012,
author = {van Iterson, M. and van Haagen, H. and Goeman, J. J.},
doi = {10.1002/pmic.201100395},
journal = {Proteomics},
number = {4-5},
pages = {543--9},
title = {{Resolving confusion of tongues in statistics and machine learning: a primer for biologists and bioinformaticians.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22246801},
volume = {12},
year = {2012}
}
@inproceedings{Wagstaff2012,
author = {Wagstaff, K. L.},
title = {{Machine Learning that Matters}},
booktitle = {Proc. Intl. Conf. Machine Learning},
year = {2012}
}
@book{Skillicorn2007,
author = {Skillicorn, D.},
editor = {Kumar, V.},
publisher = {Chapman & Hall CRC},
title = {{Understanding Complex Datasets: Data Mining with Matrix Decompositions}},
year = {2007}
}
@article{vanderMaaten2011,
author = {van der Maaten, L. and Hinton, G.},
doi = {10.1007/s10994-011-5273-4},
journal = {Machine Learning},
title = {{Visualizing non-metric similarities in multiple maps}},
url = {http://www.springerlink.com/index/10.1007/s10994-011-5273-4},
year = {2011}
}
@article{Jeong2009,
author = {Jeong, D. H. and Ziemkiewicz, C. and Fisher, B. and Ribarsky, W. and Chang, R.},
journal = {Computer Graphics Forum},
number = {3},
pages = {767--774},
title = {{iPCA: An Interactive System for PCA-based Visual Analytics}},
volume = {28},
year = {2009}
}
@article{Ingram2009,
author = {Ingram, S. and Munzner, T. and Olano, M.},
title = {{Glimmer: multilevel MDS on the GPU.}},
doi = {10.1109/TVCG.2008.85},
journal = {IEEE transactions on visualization and computer graphics},
number = {2},
pages = {249--61},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19147889},
volume = {15},
year = {2009}
}
@techreport{VanderMaaten2007,
Author = {{van der Maaten}, LJP and Postma, EO and {van den Herik}, HJ},
Institution = {Tilburg University, TiCC-TR 2009-005},
Title = {{Dimensionality reduction: A comparative review}},
Year = {2009}
}
@article{VanderMaaten2008,
author = {{van der Maaten}, Laurens and Hinton, Geoffrey},
journal = {Journ. Machine Learning Research},
pages = {2579--2605},
title = {{Visualizing data using t-SNE}},
volume = {9},
year = {2008}
}
@inproceedings{West2003,
author = {West, M.},
title = {{Bayesian factor regression models in the ``Large p, Small n'' paradigm}},
booktitle = {Bayesian Statistics},
pages = {723--732},
publisher = {Oxford University Press},
year = {2003}
}
@inproceedings{Williams2004,
author={Williams, M. and Munzner, T.},
booktitle={Proc. IEEE Symp. Information Visualization (InfoVis)},
title={Steerable, progressive multidimensional scaling},
year={2004},
pages={57--64}
}
@article{France2011,
author={France, S.L. and Carroll, J.D.},
journal={IEEE Trans. Systems, Man, and Cybernetics},
title={Two-Way Multidimensional Scaling: A Review},
year={2011},
volume={41},
number={5},
pages={644 -661},
doi={10.1109/TSMCC.2010.2078502},
}
@book{friendly2000visualizing,
title={Visualizing categorical data},
author={Friendly, M.},
isbn={9781580256605},
lccn={2001270863},
url={http://books.google.ca/books?id=eG0phz62f1cC},
year={2000},
publisher={SAS Institute}
}
@article{Chen2009a,
author = {Chen, Y. and Wang, L. and Dong, M. and Hua, J.},
doi = {10.1109/TVCG.2009.140},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {1161--8},
pmid = {19834185},
title = {{Exemplar-based visualization of large document corpus}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834185},
volume = {15},
year = {2009}
}
@article{Yang2010,
author = {Yang, J. and Luo, D. and Liu, Y.},
doi = {10.1109/MCG.2010.93},
journal = {IEEE Computer Graphics and Applications},
month = sep,
number = {5},
pages = {32--41},
title = {{Newdle: Interactive Visual Exploration of Large Online News Collections}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5556732},
volume = {30},
year = {2010}
}
@inproceedings{Cuadros2007,
author = {Cuadros, A. M. and Paulovich, F. V. and Minghim, R.},
title = {{Point placement by phylogenetic trees and its application for visual analysis of document collections}},
booktitle = {Proc. IEEE Conference on Visual Analytics Software and Technologies (VAST '07)},
url = {http://www.vgtc.org/PDF/slides/2007/vast/vast\_s04p01.pdf},
year = {2007}
}
@article{Paulovich2008,
author = {Paulovich, F. V. and Nonato, L. G. and Minghim, R. and Levkowitz, H.},
doi = {10.1109/TVCG.2007.70443},
journal = {IEEE transactions on visualization and computer graphics},
number = {3},
pages = {564--75},
title = {{Least square projection: a fast high-precision multidimensional projection technique and its application to document mapping.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18369264},
volume = {14},
year = {2008}
}
@inproceedings{Oesterling2011,
author = {Oesterling, P. and Scheuermann, G. and Teresniak, S. and Heyer, G. and Koch, S. and Ertl, T. and Weber, G. H.},
title = {{Two-stage Framework for a Topology-Based Projection and Visualization of Classified Document Collections}},
booktitle = {Proc. IEEE Conference on Visual Analytics Software and Technologies (VAST '11)},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=5652940},
year = {2011}
}
@article{Safonova2004,
address = {New York, New York, USA},
author = {Safonova, A. and Hodgins, J. K. and Pollard, N. S.},
doi = {10.1145/1186562.1015754},
journal = {Proc. ACM SIGGRAPH '04},
pages = {514},
title = {{Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces}},
url = {http://portal.acm.org/citation.cfm?doid=1186562.1015754},
year = {2004}
}
@article{Mann1998,
author = {Mann, M. E. and Bradley, R. S. and Hughes, M. K.},
journal = {Nature},
number = {6678},
pages = {779--787},
title = {{Global-scale temperature patterns and climate forcing over the past six centuries}},
url = {http://www.astro.uu.nl/~werkhvn/study/Y3\_05\_06/data/talk/14-juni/mannetal1998.pdf},
volume = {392},
year = {1998}
}
@techreport{Marks1997,
author = {Marks, J. and Andalman, B. and Beardsley, P. A. and Freeman, W. and Gibson, S. and Hodgins, J. and Kang, T. and Mirtich, B. and Pfister, H. and Ruml, W. and Kyall, K. and Seims, J. and Shieber, S.},
title = {{Design galleries: A general approach to setting parameters for computer graphics and animation}},
institution = {Mitsubishi Electric Research Laboratories},
pages = {389--400},
publisher = {ACM Press/Addison-Wesley Publishing Co.},
url = {http://dl.acm.org/citation.cfm?id=258887},
year = {1997}
}
@article{Grochow2004,
author = {Grochow, K. and Martin, S. L. and Hertzmann, A. and Popovi\'{c}, Z.},
file = {:Users/matthewbrehmer/Documents/Academics/Mendeley/Grochow et al/Grochow et al. - 2004 - Style-based inverse kinematics.pdf:pdf},
number = {3},
pages = {522--531},
publisher = {ACM},
title = {{Style-based inverse kinematics}},
url = {http://portal.acm.org/citation.cfm?id=1015706.1015755},
volume = {23},
year = {2004}
}
@inproceedings{Elad2001,
author = {Elad, A. and Kimmel, R.},
title = {{Bending Invariant Representations for Surfaces}},
booktitle = {Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on},
pages = {I--168},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=990472},
volume = {1},
year = {2001}
}
@inproceedings{Ward1994,
author = {Ward, M. O.},
title = {{XmdvTool : Integrating Multiple Met hods for Visualizing Multivariate Data N-Dimensional Data Visualization}},
booktitle = {Proc. IEEE Conf. Visualization (Vis '94)},
pages = {326--333},
year = {1994}
}
@article{Guo2003,
author = {Guo, D.},
doi = {10.1057/palgrave.ivs.9500053},
journal = {Information Visualization},
month = dec,
number = {4},
pages = {232--246},
title = {{Coordinating computational and visual approaches for interactive feature selection and multivariate clustering}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500053},
volume = {2},
year = {2003}
}
@article{Peng2004,
author = {Peng, W. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Clutter Reduction in Multi-Dimensional Data Visualization Using Dimension Reordering}},
doi = {10.1109/INFVIS.2004.15},
journal = {Proc. IEEE Symposium on Information Visualization (INFOVIS' 04)},
pages = {89--96},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382895},
year = {2004}
}
@inproceedings{Ankerst1998,
author = {Ankerst, M. and Berchtold, S. and Keim, D. A.},
title = {{Similarity clustering of dimensions for an enhanced visualization of multidimensional data}},
booktitle = {Proc. IEEE Symposium on Information Visualization (INFOVIS' 98)},
pages = {52--60},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=729559},
year = {1998}
}
@inproceedings{Chalmers1996,
author = {Chalmers, M.},
title = {{A Linear Iteration Time Layout Algorithm for Visualising High-Dimensional Data}},
booktitle = {Proc. IEEE Visualization '96},
pages = {127--131},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=567787},
year = {1996}
}
@inproceedings{Yang2003,
author = {Yang, J. and Ward, M. O. and Rundensteiner, E. A. and Huang, S.},
title = {{Visual hierarchical dimension reduction for exploration of high dimensional datasets}},
booktitle = {Proc. IEEE Visualization (VisSym '03)},
pages = {19--28},
url = {http://dl.acm.org/citation.cfm?id=769924},
volume = {1},
year = {2003}
}
@inproceedings{Yang2004,
author = {Yang, J. and Patro, A. and Huang, S. and Mehta, N. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Value and Relation Display for Interactive Exploration of High Dimensional Datasets}},
booktitle = {Proc. IEEE Symposium on Information Visualization (INFOVIS' 04)},
pages = {73--80},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=1382893},
year = {2004}
}
@inproceedings{Yang2003,
author = {Yang, J. and Peng, W. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Interactive Hierarchical Dimension Ordering, Spacing and Filtering for Exploration of High Dimensional Datasets}},
booktitle = {IEEE Symposium on Information Visualization (INFOVIS '03).},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=1249015},
year = {2003}
}
@article{Reveret2005,
address = {New York, New York, USA},
author = {Reveret, L. and Favreau, L. and Depraz, C. and Cani, M. P.},
doi = {10.1145/1073368.1073386},
journal = {Proceedings of the 2005 ACM SIGGRAPH/Eurographics symposium on Computer animation - SCA '05},
pages = {135},
publisher = {ACM Press},
title = {{Morphable model of quadrupeds skeletons for animating 3D animals}},
url = {http://portal.acm.org/citation.cfm?doid=1073368.1073386},
year = {2005}
}
@article{Bronstein2006,
author = {Bronstein, A. M. and Bronstein, M. M. and Kimmel, R.},
doi = {10.1073/pnas.0508601103},
journal = {Proceedings of the National Academy of Sciences of the United States of America},
number = {5},
pages = {1168--72},
title = {{Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching.}},
url = {http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1360551\&tool=pmcentrez\&rendertype=abstract},
volume = {103},
year = {2006}
}
@article{Bronstein2006a,
author = {Bronstein, A. and Bronstein, M. and Bruckstein, A. and Kimmel, R.},
journal = {Articulated Motion and Deformable Objects},
pages = {48--57},
publisher = {Springer},
title = {{Matching two-dimensional articulated shapes using generalized multidimensional scaling}},
url = {http://www.springerlink.com/index/RT435U315728622N.pdf},
year = {2006}
}
@article{Keller,
author = {Keller, F.},
title = {{Clustering: Connectionist and Statistical Language Processing}}
}
@article{Verma2011,
author = {Verma, A.},
title = {{Classification Vs Clustering}},
journal = {The Social Canvas: Dashboard of Life},
url = {http://thesocialcanvas.blogspot.com/2011/03/classification-vs-clustering.html},
year = {2011}
}
@article{Rubinov2006,
author = {Rubinov, A. M. and Soukhorokova, N. V. and Ugon, J.},
doi = {10.1016/j.ejor.2005.04.047},
journal = {European Journal of Operational Research},
number = {3},
pages = {849--865},
title = {{Classes and clusters in data analysis}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0377221705006867},
volume = {173},
year = {2006}
}
@article{Johansson2009,
author = {Johansson, S. and Johansson, J.},
doi = {10.1109/TVCG.2009.153},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {993--1000},
title = {{Interactive dimensionality reduction through user-defined combinations of quality metrics.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834164},
volume = {15},
year = {2009}
}
@article{Buja2002,
author = {Buja, A. and Swayne, D. F.},
doi = {10.1007/s00357-001-0031-0},
journal = {Journal of Classification},
pages = {7--43},
title = {{Visualization Methodology for Multidimensional Scaling}},
volume = {19},
year = {2002}
}
@incollection{Holbrey2006,
author = {Holbrey, R.},
title = {{Column Reduction Techniques}},
booktitle = {Dimension reduction algorithms for data mining and visualization},
chapter = {4},
doi = {10.1.1.124.5567},
editor = {Holbrey, R.},
pages = {31-51},
publisher = {Citesee},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.5567\&rep=rep1\&type=pdf},
year = {2006}
}
@book{Holbrey2006,
author = {Holbrey, R.},
pages = {1--113},
title = {{Dimension reduction algorithms for data mining and visualization}},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.124.5567\&rep=rep1\&type=pdf},
year = {2006}
}
@incollection{Munzner2011,
author = {Munzner, T.},
title = {{Attribute Reduction Methods}},
booktitle = {Information Visualization: Principles, Methods, and Practice},
chapter = {8},
edition = {Draft},
year = {2011}
}
@book{Tan2006,
address = {Boston, Massachusetts, USA},
author = {Tan, P. N. and Steinbach, M. and Kumar, V.},
isbn = {0-321-32136-7},
pages = {769},
publisher = {Pearson Education, Inc.},
title = {{Introduction to Data Mining}},
year = {2006}
}
@inproceedings{Ingram2010,
author = {Ingram, S. and Munzner, T. and Irvine, V. and Tory, M. and Bergner, S. and Torsten, M.},
title = {{DimStiller: Workflows for dimensional analysis and reduction}},
booktitle = {Proc. IEEE Conference on Visual Analytics Software and Technologies (VAST '10)},
pages = {3--10},
year = {2010}
}
@article{Tenenbaum2000,
author = {Tenenbaum, J. B. and de Silva, V. and Langford, J .C.},
title = {{A global geometric framework for nonlinear dimensionality reduction.}},
doi = {10.1126/science.290.5500.2319},
journal = {Science (New York, N.Y.)},
number = {5500},
pages = {2319--23},
url = {http://www.ncbi.nlm.nih.gov/pubmed/11125149},
volume = {290},
year = {2000}
}
@article{Matusik2003,
author = {Matusik, W. and Pfister, H. and Brand, M. and McMillan, L.},
title = {{A data-driven reflectance model}},
doi = {10.1145/882262.882343},
journal = {ACM Transactions on Graphics},
number = {3},
pages = {759},
url = {http://portal.acm.org/citation.cfm?doid=882262.882343},
volume = {22},
year = {2003}
}
@article {Young1938,
author = {Young, G. and Householder, A.},
title = {Discussion of a set of points in terms of their mutual distances},
journal = {Psychometrika},
publisher = {Springer},
pages = {19--22},
volume = {3},
issue = {1},
year = {1938}
}
@article {Torgerson1952,
author = {Torgerson, W. S.},
title = {Multidimensional scaling: I. Theory and method},
journal = {Psychometrika},
publisher = {Springer},
pages = {401--419},
volume = {17},
issue = {4},
year = {1952}
}
@inproceedings{Belkin2001,
Author = {Belkin, Mikhail and Niyogi, Partha},
Title = {{Laplacian eigenmaps and spectral techniques for embedding and clustering}},
Booktitle = {Proc. Advances in Neural Information Processing Systems (NIPS)},
publisher = {MIT Press},
pages = {585--591},
Year = {2001},
Bdsk-Url-1 = {http://www-2.cs.cmu.edu/Groups/NIPS/NIPS2001/papers/psgz/AA42.ps.gz}
}
@article{balasubramanian2002,
title={The {I}somap Algorithm and Topological Stability},
author={Balasubramanian, M. and Schwartz, E.L.},
journal={Science},
volume={295},
pages={7},
year={2002},
publisher={American Association for the Advancement of Science}
}
@incollection{Cunningham2008,
author = "Cunningham, P.",
year = 2008,
title = "Dimension reduction",
booktitle = "Machine Learning Techniques for Multimedia: Case Studies on Organization and Retrieval",
publisher = "Springer",
pages = "91--112",
}
@article{jain2000,
title={Statistical pattern recognition: A review},
author={Jain, A.K. and Duin, R.P.W. and Mao, J.},
journal={IEEE Trans. Pattern Analysis and Machine Intelligence},
volume={22},
number={1},
pages={4--37},
year={2000}
}
}}}
{{{
@article{Tri-Council2010,
author = {CIHR, NSERC, SSHRC},
title = {{Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans}},
year = {2010}
}
@book{Freeman2009,
author = {Freeman, M. and Mathison, S.},
publisher = {The Guilford Press},
title = {{Researching children's experiences}},
url = {http://books.google.com/books?hl=en\&lr=\&id=QQimUOr3AYAC\&oi=fnd\&pg=PA1\&dq=Researching+Children\%E2\%80\%99s+Experiences\&ots=JRg-U5fFWR\&sig=bG8Os8dSoOJ15prHT3jTdaRrtfw},
year = {2009}
}
@article{Mathison2009,
author = {Mathison, S.},
title = {{Seeing is believing: The credibility of image-based research and evaluation}},
journal = {Applied Research and Evaluation},
url = {http://scholar.google.com/scholar?hl=en\&btnG=Search\&q=intitle:Seeing+Is+Believing:+The+Credibility+of+Image+Based+Research+and+Evaluation\#0},
year = {2009}
}
@article{Mathison1988,
author = {Mathison, S.},
title = {{Why Triangulate?}},
doi = {10.2307/1174583},
journal = {Educational Researcher},
number = {2},
pages = {13},
url = {http://links.jstor.org/sici?sici=0013-189X\%28198803\%2917\%3A2\%3C13\%3AWT\%3E2.0.CO\%3B2-L\&origin=crossref},
volume = {17},
year = {1988}
}
@article{Baff1997,
author = {Baff, S. J.},
doi = {10.1177/107780049700300407},
journal = {Qualitative Inquiry},
number = {4},
pages = {468--490},
title = {{Realism and Naturalism and Dead Dudes: Talking About Literature in 11th-Grade English}},
url = {http://qix.sagepub.com/cgi/doi/10.1177/107780049700300407},
volume = {3},
year = {1997}
}
@article{Becker1957,
author = {Becker, H. S. and Geer, B.},
journal = {Human Organization},
number = {3},
pages = {28--32},
title = {{Field Methods and Techniques Participant Observation and Interviewing: A Comparison}},
volume = {16},
year = {1957}
}
@book{Crotty1998,
author = {Crotty, M.},
publisher = {Sage Publications},
title = {{The Foundations of Social Research}},
year = {1998}
}
@incollection{Fontana1994,
author = {Fontana, A. and Frey, J.},
title = {{Interviewing: The Art of Science}},
booktitle = {The Handbook of Qualitative Research},
edition = {3rd},
pages = {361--376},
publisher = {Sage Publications},
year = {1994}
}
@article{KadriZald2004,
author = {{Kadri Zald}, J.},
journal = {City and Community},
number = {1},
pages = {29--41},
title = {{Faces of the Homeless: A Photo Essay}},
volume = {3},
year = {2004}
}
@book{Pascoe2007,
author = {Pascoe, C. J.},
publisher = {University of California Press},
title = {{Dude, You're a Fag: Masculinity and Sexuality in High School}},
year = {2007}
}
@article{Prown1982,
author = {Prown, J. D.},
journal = {Winterthur Portfolio},
number = {1},
pages = {1--19},
title = {{Mind in Matter: An Introduction to Material Culture Theory and Method}},
volume = {17},
year = {1982}
}
@article{Ryan2003,
author = {Ryan, G. W. and Bernard, H. R.},
doi = {10.1177/1525822X02239569},
journal = {Field Methods},
number = {1},
pages = {85--109},
title = {{Techniques to Identify Themes}},
url = {http://fmx.sagepub.com/cgi/doi/10.1177/1525822X02239569},
volume = {15},
year = {2003}
}
@article{Sandelowski1998,
author = {Sandelowski, M.},
journal = {Research in Nursing and Health},
pages = {375--82},
title = {{Writing a Good Read: Strategies for Re-Presenting Qualitative Data}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/9679813},
volume = {21},
year = {1998}
}
@book{Silverman2007,
author = {Silverman, D.},
publisher = {Sage Publications},
title = {{A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research}},
year = {2007}
}
@article{Krueger2002,
author = {Krueger, R. A.},
title = {{Designing and Conducting Focus Group Interviews}},
year = {2002}
}
}}}
{{{
@article{Sedlmair2012,
author = {Sedlmair, M. and Tatu, A. and Munzner, T. and Tory, M.},
title = {A Taxonomy of Visual Cluster Separation Factors},
journal ={Computer Graphics Forum (Proc. EuroVis)},
volume = {31},
number = {3},
pages = {1335--1344},
year = {2012}
}
@inproceedings{Tatu2010,
author = {Tatu, A. and Bak, P. and Bertini, E. and Keim, D. and Schneidewind, J.},
title = {{Visual quality metrics and human perception}},
booktitle = {Proc. ACM Advanced Visual Interfaces (AVI)},
doi = {10.1145/1842993.1843002},
pages = {49},
url = {http://portal.acm.org/citation.cfm?doid=1842993.1843002},
year = {2010}
}
@article{Pretorius2009,
author = {Pretorius, A. J. and {Van Wijk}, J. J.},
doi = {10.1057/ivs.2009.13},
journal = {Information Visualization},
number = {3},
pages = {153--166},
title = {{What does the user want to see? What do the data want to be?}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2009.13},
volume = {8},
year = {2009}
}
@article{Wilkinson2008,
author = {Wilkinson, L. and Wills, G.},
doi = {10.1198/106186008X320465},
journal = {Computational and Graphical Statistics},
number = {2},
pages = {473--491},
title = {{Scagnostics Distributions}},
url = {http://www.tandfonline.com/doi/abs/10.1198/106186008X320465},
volume = {17},
year = {2008}
}
@article{Ahuja1989,
author = {Ahuja, N. and Tuceryan, M.},
doi = {10.1016/0734-189X(89)90146-1},
journal = {Computer Vision, Graphics, and Image Processing},
number = {3},
pages = {304--356},
title = {{Extraction of early perceptual structure in dot patterns: Integrating region, boundary, and component gestalt}},
url = {http://linkinghub.elsevier.com/retrieve/pii/0734189X89901461},
volume = {48},
year = {1989}
}
@techreport{Bigelow2012,
author = {Bigelow, A.},
title = {{Driving Genetics With Experimental Visualization}},
institution = {Scientific Computing and Imaging Institute, University of Utah},
pages = {10},
url = {http://sci.utah.edu/~abigelow/Papers/BS Thesis/Thesis.pdf},
year = {2012}
}
@article{Lee2005a,
author = {Lee, E. K. and Cook, D. and Klinke, S. and Lumley, T.},
doi = {10.1198/106186005X77702},
journal = {Computational and Graphical Statistics},
month = dec, = {4},
pages = {831--846},
title = {{Projection Pursuit for Exploratory Supervised Classification}},
url = {http://www.tandfonline.com/doi/abs/10.1198/106186005X77702},
volume = {14},
year = {2005}
}
@techreport{Majumder2010,
author = {Majumder, M. and Hofmann, H. and Cook, D.},
institution = {Department of Statistics, Iowa State University},
pages = {1--6},
title = {{Visual Statistical Inference for Regression Parameters}},
year = {2010}
}
@techreport{Zhao2012,
author = {Zhao, Y. and Cook, D. and Hofmann, H. and Majumder, M. and Chowdhury, N. R.},
institution = {Department of Statistics, Iowa State University},
pages = {1--25},
title = {{Mind Reading Using an Eyetracker to See How People Are Looking at Lineups}},
year = {2012}
}
@article{Brentnall2011,
author = {Brentnall, A. R. and Duffy, S. W. and Crowder, M. J. and Gillan, M. G. C. and Astley, S. M. and Wallis, M. G. and James, J. and Boggis, C. R. M. and Gilbert, F. J.},
doi = {10.1214/11-AOAS481},
journal = {Annals of Applied Statistics},
number = {4},
pages = {2448--2469},
title = {{A method for exploratory repeated-measures analysis applied to a breast-cancer screening study}},
url = {http://projecteuclid.org/euclid.aoas/1324399602},
volume = {5},
year = {2011}
}
@article{Gelman2012,
author = {Gelman, A. and Unwin, A.},
journal = {Computational and Graphical Statistics},
number = {June},
pages = {1--31},
title = {{Infovis and Statistical Graphics: Different Goals, Different Looks}},
year = {2012}
}
@article{Gelman2004,
author = {Gelman, A.},
doi = {10.1198/106186004X11435},
journal = {Computational and Graphical Statistics},
number = {4},
pages = {755--779},
title = {{Exploratory Data Analysis for Complex Models}},
url = {http://www.tandfonline.com/doi/abs/10.1198/106186004X11435},
volume = {13},
year = {2004}
}
@article{Langsrud2005,
author = {Langsrud, �urnal = {Statistics and Computing},
pages = {53--60},
title = {{Rotation tests}},
volume = {15},
year = {2005}
}
@article{Tukey1965,
author = {Tukey, J. W.},
journal = {The American Statistician},
number = {2},
pages = {23--28},
title = {{The Technical Tools of Statistics}},
volume = {19},
year = {1965}
}
@article{Pitman1937,
author = {Pitman, E. J. G.},
journal = {Royal Statistical Society},
pages = {121--124},
title = {{Significance Tests Which May Be Applied to Samples From Any Population}},
volume = {4},
year = {1937}
}
@article{Buja2009,
author = {Buja, A. and Cook, D. and Hofmann, H. and Lawrence, M. and Lee, E. K. and Swayne, D. F. and Wickham, H.},
doi = {10.1098/rsta.2009.0120},
journal = {Royal Society Philosophical Transactions A},
number = {1906},
pages = {4361--4383},
title = {{Statistical inference for exploratory data analysis and model diagnostics}},
volume = {367},
year = {2009}
}
@article{Wolfe2010,
author = {Wolfe, J. M. and Wert, M. J. V.},
doi = {10.1016/j.cub.2009.11.066},
journal = {Current Biology},
number = {2},
pages = {121--124},
publisher = {Elsevier Ltd},
title = {{Report Varying Target Prevalence Reveals Two Dissociable Decision Criteria in Visual Search}},
url = {http://dx.doi.org/10.1016/j.cub.2009.11.066},
volume = {20},
year = {2010}
}
@article{Wickham2010,
author = {Wickham, H. and Cook, D. and Hofmann, H. and Buja, A.},
title = {{Graphical inference for infovis}},
doi = {10.1109/TVCG.2010.161},
journal = {IEEE Trans. Visualization Computer Graphics (TVCG)},
number = {6},
pages = {973--9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975134},
volume = {16},
year = {2010}
}
@article{Wickham2011a,
author = {Wickham, H.},
title = {{Ggplot2}},
doi = {10.1002/wics.147},
journal = {Wiley Interdisciplinary Reviews: Computational Statistics},
number = {2},
pages = {180--185},
url = {http://doi.wiley.com/10.1002/wics.147},
volume = {3},
year = {2011}
}
@article{Wood2011,
author = {Wood, J. and Badawood, D. and Dykes, J. and Slingsby, A.},
title = {{BallotMaps: Detecting name bias in alphabetically ordered ballot papers}},
doi = {10.1109/TVCG.2011.174},
journal = {IEEE Trans. Visualization Computer Graphics (TVCG)},
number = {12},
pages = {2384--2391},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034359},
volume = {17},
year = {2011}
}
@inproceedings{Kairam2012,
author = {Kairam, S. and Maclean, D. and Savva, M. and Heer, J.},
title = {{GraphPrism: Compact visualization of network structure}},
booktitle = {Proc. Conf. Advanced Visual Interfaces (AVI)},
year = {2012}
}
@inproceedings{Ziemkiewicz2012,
author = {Ziemkiewicz, C. and Gomez, S. and Laidlaw, D. H.},
title = {{Analysis within and between graphs: Observed user strategies in immunobiology visualization}},
booktitle = {Proc. SIGCHI conf. Human factors in computing systems (CHI)},
year = {2012}
}
@inproceedings{Willett2012,
author = {Willett, W. and Heer, J. and Agrawala, M.},
title = {{Strategies for Crowdsourcing Social Data Analysis}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Kosara2010,
author = {Kosara, R. and Ziemkiewicz, C.},
title = {{Do Mechanical Turks Dream of Square Pie Charts?}},
booktitle = {Proc. of ~BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV '10)},
year = {2010}
}
@inproceedings{Heer2010,
author = {Heer, J. and Bostock, M.},
title = {{Crowdsourcing graphical perception: using mechanical turk to assess visualization design}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '10)},
pages = {203--212},
url = {http://dl.acm.org/citation.cfm?id=1753357},
year = {2010}
}
}}}
{{{
@article{Gan2014,
author = {Gan, Q. and Zhu, M. and Li, M. and Liang, T. and Cao, Y. and Zhou, B.},
title = {{Document visualization: An overview of current research}},
journal = {Wiley Interdisciplinary Reviews: Computational Statistics},
number = {1},
pages = {19--36},
volume = {6},
year = {2014}
}
@inproceedings{Alexander2014,
author = {Alexander, E. and Kohlmann, J. and Valenza, R. and Witmore, M. and Gleicher, M.},
title = {{Serendip: Topic model-driven visual exploration of text corpora}},
booktitle = {To appear in IEEE VAST},
year = {2014}
}
@article{Cui2014,
author = {Cui, W. and Liu, S. and Wu, Z. and Wei, H.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{How hierarchical topics evolve in large text corpora}},
year = {2014}
}
@techreport{Sedlmair2012a,
author = {Sedlmair, M. and Brehmer, M. and Ingram, S. and Munzner, T.},
institution = {Dept. Computer Science, University of British Columbia},
title = {{Dimensionality reduction in the wild: Gaps and guidance}},
year = {2012}
}
@article{Hetzler2004,
author = {Hetzler, E. and Turner, A.},
journal = {IEEE Computer Graphics \& Applications (CG\&A)},
number = {5},
pages = {22-26},
title = {{Analysis experiences using information visualization}},
volume = {24},
year = {2004}
}
@article{Keller2012,
title = {{Own a gun? Tell us why}},
author = {Keller, M.},
journal = {The Daily Beast},
year = {Dec. 22, 2012},
note = {http://goo.gl/2yRgk}}
}
@article{Playford2013,
title = {{For their eyes only: Police misconduct hidden from public by secrecy law, weak oversight}},
author = {Peddie, S. and Playford, A.},
journal = {Newsday},
year = {Dec. 28, 2013},
note = {http://goo.gl/KBT7CA}}
}
@article{Keller2012a,
title = {{The other month, we asked readers why they did or did not own guns}},
author = {Keller, M.},
journal = {NewsBeast Labs},
year = {Feb. 12, 2013},
note = {http://goo.gl/jKBdeH}}
}
@article{Wade2012a,
title = {{How the {\it Tulsa World} used {\it AP}'s Overview to report on 8,000 police department emails}},
author = {Wade, J.},
journal = {PBS Idea Lab},
year = {Jun. 26, 2012},
note = {http://goo.gl/QJ64R}}
}
@article{Gillum2012,
title = {{Ryan asked for federal help as he championed cuts}},
author = {Gillum, J.},
journal = {Associated Press},
year = {Oct. 12, 2012},
note = {http://goo.gl/RM7uk}}
}
@article{Wade2012,
title = {{TPD working through flawed mobile system}},
author = {Wade, J.},
journal = {Tulsa World},
year = {Jun. 3, 2012},
note = {http://goo.gl/yE98e2}}
}
@article{Wade2012a,
title = {{How the Tulsa World used AP's Overview to report on 8,000 police department emails}},
author = {Wade, J.},
journal = {PBS Idea Lab},
year = {Jun. 26, 2012},
note = {http://goo.gl/QJ64R}}
}
@article{Howard2012,
author = {Howard, A.},
title = {{Profiling data journalists}},
journal = {O'Reilly Strata: Making Data Work},
year = {2012},
note = {Mar. http://goo.gl/64Yh2}}
}
@article{Stray2012a,
author = {Stray, J.},
title={{Using Overview to analyze 4500 pages of documents on security contractors in Iraq}},
journal = {Associated Press},
year = {Feb. 21, 2012},
note={http://goo.gl/6SmNc}}
}
@article{Stray2012,
author = {Stray, J.},
title={{What did private security contractors do in Iraq?}},
journal = {Associated Press},
year = {Feb. 21, 2012},
note={http://goo.gl/qqGGh}}
}
@article{Stray2010,
author = {Stray, J.},
title={{A full-text visualization of the Iraq War Logs}},
journal = {Associated Press},
year = {Dec. 10, 2010},
note={http://goo.gl/IITFpE}}
}
@inproceedings{Amar2005,
author = {Amar, R. and Eagan, J. and Stasko, J. T.},
booktitle = {Proc. IEEE Conf. Information Visualization},
pages = {111--117},
title = {{Low-level components of analytic activity in information visualization}},
year = {2005}
}
@inproceedings{Amar2004,
author = {Amar, R. and Stasko, J. T.},
booktitle = {Proc. IEEE Symp. Information Visualization},
pages = {143--149},
title = {{A knowledge task-based framework for design and evaluation of information visualizations}},
year = {2004}
}
@article{Anderson2010,
author = {Anderson, C. W.},
journal = {Political Communication},
number = {3},
pages = {289--309},
title = {{Journalistic networks and the diffusion of local news: The brief, happy news life of the ⡮cisville Four� volume = {27},
year = {2010}
}
@article{Anderson2011,
author = {Anderson, C. W.},
journal = {Intl. J. Communication},
pages = {529--547},
title = {{Deliberative, agonistic, and algorithmic audiences: Journalism's vision of its public in an age of audience transparency}},
volume = {5},
year = {2011}
}
@article{Anderson2013,
author = {Anderson, C. W. and Kreiss, D.},
pages = {1--24},
title = {{Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism}},
year = {2013}
}
@inproceedings{Andre2009,
author = {Andr\'{e}, P. and Teevan, J. and Dumais, S. T.},
booktitle = {Proc. ACM Conf. Creativity \& Cognition},
pages = {305--314},
title = {{Discovery Is never by chance: Designing for (un)serendipity}},
year = {2009}
}
@book{Andrienko2006,
author = {Andrienko, G. and Andrienko, N.},
pages = {715},
publisher = {Springer-Verlag},
title = {{Exploratory Analysis of Spatial and Temporal Data: A Systematic Approach}},
year = {2006}
}
@book{Bailey1994,
author = {Bailey, K. D.},
publisher = {SAGE Publications, Inc},
title = {{Typologies and Taxonomies: An Introduction to Classification}},
year = {1994}
}
@inproceedings{Beaudouin-Lafon2004,
author = {Beaudouin-Lafon, M.},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {15--22},
publisher = {ACM},
title = {{Designing Interaction, not Interfaces}},
year = {2004}
}
@article{Becker1957,
author = {Becker, H. S. and Geer, B.},
journal = {Human Organization},
number = {3},
pages = {28--32},
title = {{Field methods and techniques participant observation and interviewing: A comparison}},
volume = {16},
year = {1957}
}
@article{Brehmer2013,
author = {Brehmer, M. and Munzner, T.},
number = {12},
pages = {2376--2385},
title = {{A multi-level typology of abstract visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
volume = {19},
year = {2013}
}
@article{Sedlmair2013,
author = {Sedlmair, M. and Munzner, T. and Tory, M.},
number = {12},
pages = {2634--2643},
title = {{Empirical guidance on scatterplot and dimension reduction technique choices}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
volume = {19},
year = {2013}
}
@inproceedings{Budiu2009,
author = {Budiu, R. and Pirolli, P. and Hong, L.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {615--624},
title = {{Remembrance of things tagged: How tagging effort affects tag production and human memory}},
year = {2009}
}
@incollection{Carpendale2008,
address = {Berlin},
author = {Carpendale, S.},
booktitle = {Information Visualization: Human-Centered Issues and Perspectives},
chapter = {2},
editor = {Kerren, A. and Stasko, J. T. and Fekete, J. D. and North, C.},
pages = {19--45},
publisher = {Springer},
title = {{Evaluating information visualizations}},
year = {2008}
}
@article{Chang2009,
author = {Chang, R. and Ziemkiewicz, C. and Green, T. M. and Ribarsky, W.},
journal = {IEEE Computer Graphics and Applications (CG\&A)},
number = {2},
pages = {14--7},
title = {{Defining insight for visual analytics}},
volume = {29},
year = {2009}
}
@inproceedings{Chang2010,
author = {Chang, R. and Ziemkiewicz, C. and Pyzh, R. and Kielman, J. and Ribarsky, W.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Learning-based evaluation of visual analytic systems}},
year = {2010}
}
@book{Charmaz2006,
author = {Charmaz, K.},
publisher = {Sage Publications, Inc.},
title = {{Constructing Grounded Theory: A Practical Guide through Qualitative Analysis.}},
year = {2006}
}
@article{Chen2009a,
author = {Chen, Y. and Wang, L. and Dong, M. and Hua, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1161--1168},
pmid = {85},
title = {{Exemplar-based visualization of large document corpus}},
volume = {15},
year = {2009}
}
@article{Chuang2012,
author = {Chuang, J. and Manning, C. D. and Heer, J.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {3},
pages = {1--29},
title = {{鴨out the clutter of unimportant words턥scriptive keyphrases for text visualization}},
volume = {19},
year = {2012}
}
@inproceedings{Chuang2012a,
author = {Chuang, J. and Ramage, D. and Manning, C. D. and Heer, J.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {443--452},
title = {{Interpretation and trust: Designing model-driven visualizations for text analysis}},
year = {2012}
}
@article{Collins2009,
author = {Collins, C. and Carpendale, S. and Penn, G.},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {1039--1046},
title = {{DocuBurst: Visualizing document content using language structure}},
volume = {28},
year = {2009}
}
@inproceedings{Correll2013,
author = {Correll, M. A. and Alexander, E..C. and Gleicher, M.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
title = {{Quantity estimation in visualizations of tagged text}},
year = {2013}
}
@article{Crestani2006,
author = {Crestani, F. and Wu, S.},
journal = {Information Processing \& Management},
number = {5},
pages = {1137--1150},
title = {{Testing the cluster hypothesis in distributed information retrieval}},
volume = {42},
year = {2006}
}
@book{Crotty1998,
author = {Crotty, M.},
publisher = {Sage Publications, Inc.},
title = {{The Foundations of Social Research}},
year = {1998}
}
@inproceedings{Cuadros2007,
author = {Cuadros, A. M. and Paulovich, F. V. and Minghim, R.},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
title = {{Point placement by phylogenetic trees and its application for visual analysis of document collections}},
year = {2007}
}
@inproceedings{Diakopoulos2012,
author = {Diakopoulos, N. and {De Choudhury}, M. and Naaman, M.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {2451},
publisher = {ACM Press},
title = {{Finding and assessing social media information sources in the context of journalism}},
year = {2012}
}
@inproceedings{Eisenstein2012,
author = {Eisenstein, J. and Chau, D. H. P. and Kittur, A. and Xing, E. P.},
booktitle = {Proc. Extended Abstracts ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {2177--2182},
title = {{TopicViz: Interactive topic exploration in document collections}},
year = {2012}
}
@inproceedings{Ellis2006,
author = {Ellis, G. and Dix, A.},
booktitle = {Proc. ACM BELIV Workshop},
pages = {1},
title = {{An explorative analysis of user evaluation studies in information visualisation}},
year = {2006}
}
@inproceedings{Endert2012,
author = {Endert, A. and Fiaux, P. and North, C.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
publisher = {ACM},
pages = {473--482},
title = {{Semantic interaction for visual text analytics}},
year = {2012}
}
@article{Endert2012b,
author = {Endert, A. and Fiaux, P. and North, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {2879--2888},
title = {{Semantic interaction for sensemaking: Inferring analytical reasoning for model steering}},
volume = {18},
year = {2012}
}
@inproceedings{Endert2012a,
author = {Endert, A. and Fox, S. and Maiti, D. and Leman, S. and North, C.},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {555--562},
title = {{The semantics of clustering: analysis of user-generated spatializations of text documents}},
year = {2012}
}
@incollection{Fontana1994,
author = {Fontana, A. and Frey, J.},
booktitle = {The Handbook of Qualitative Research},
edition = {3rd editio},
pages = {361--376},
publisher = {Thousand Oaks: Sage Publications},
title = {{Interviewing: The art of science}},
year = {1994}
}
@inproceedings{Gonzalez2003,
author = {Gonzalez, V. M and Kobsa, A.},
booktitle = {Proc. Intl. Conf. Information Visualization},
pages = {331--336},
title = {{Benefits of information visualization systems for administrative data analysts}},
year = {2003}
}
@inproceedings{Gonzalez2003a,
author = {Gonzalez, V. M and Kobsa, A.},
booktitle = {Proc. Intl. Conf. Knowledge Management},
pages = {92--102},
title = {{A workplace study of the adoption of information visualization systems}},
year = {2003}
}
@inproceedings{Goyal2013,
author = {Goyal, N. and Leshed, G. and Fussell, S. R.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {2721--2724},
title = {{Effects of visualization and note-taking on sensemaking and analysis}},
year = {2013}
}
@article{Grimmer2011,
author = {Grimmer, J. and King, G.},
journal = {Proc. National Academy of Sciences (PNAS)},
number = {7},
pages = {2643--2650},
title = {{General purpose computer-assisted clustering and conceptualization.}},
volume = {108},
year = {2011}
}
@article{Hopkins2010,
author = {Hopkins, D. J. and King, G.},
journal = {American J. Political Science},
number = {1},
pages = {229--247},
title = {{A Method of Automated Nonparametric Content Analysis for Social Science}},
volume = {54},
year = {2010}
}
@inproceedings{Ward2005,
author = {Huang, S. and Ward, M. O. and Rundensteiner, E. A.},
booktitle = {Proc. IEEE Intl. Conf. Coordinated and Multiple Views in Exploratory Visualization (CMV)},
pages = {63--74},
title = {{Exploration of dimensionality reduction for text visualization}},
year = {2005}
}
@article{Ingram2009,
author = {Ingram, S. and Munzner, T. and Olano, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {249--261},
title = {Glimmer: Multilevel {MDS} on the {GPU}.},
volume = {15},
year = {2009}
}
@techreport{Ingram2012,
author = {Ingram, S. and Stray, J. and Munzner, T.},
institution = {University of British Columbia},
title = {{Hierarchical clustering and tagging of mostly disconnected data}},
year = {2012}
}
@article{Isenberg2009,
author = {Isenberg, P. and Fisher, D.},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {1031--1038},
title = {{Collaborative brushing and linking for co-located visual analytics of document collections}},
volume = {28},
year = {2009}
}
@inproceedings{Isenberg2008a,
author = {Isenberg, P. and Tang, A. and Carpendale, S.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
title = {{An exploratory study of visual information analysis}},
pages = {1217-1226 },
year = {2008}
}
@inproceedings{Isenberg2008,
author = {Isenberg, P. and Zuk, T. and Collins, C. and Carpendale, S.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Grounded evaluation of information visualizations}},
year = {2008}
}
@article{Jenkins2012,
author = {Jenkins, P.},
keywords = {cognitive biases,cross-cultural perspectives,reputation for resolve},
title = {{Diplomatic Cables and Reputation for Resolve}},
volume = {746},
year = {2012}
}
@inproceedings{Jianu2012,
author = {Jianu, R. and Laidlaw, D. H.},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {2953--2962},
title = {{An evaluation of how small user interface changes can improve scientistsᮡlytic strategies}},
year = {2012}
}
@inproceedings{Kang2011,
author = {Kang, Y. A. and Stasko, J. T.},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {19--28},
title = {{Characterizing the intelligence analysis process: Informing visual analytics design through a longitudinal field study}},
year = {2011}
}
@article{Keim2013,
author = {Keim, D. A. and Krstajic, M. and Rohrdantz, C. and Schreck, T.},
journal = {IEEE Computer},
title = {{Real-time visual analytics of text data streams}},
volume = {To appear},
year = {2013}
}
@unpublished{King2012,
author = {King, G. and Pan, J. and Roberts, M.},
institution = {Harvard University},
number = {917},
title = {{How censorship in China allows government criticism but silences collective expression}},
year = {2012}
}
@article{Lam2012,
author = {Lam, H. and Bertini, E. and Isenberg, P. and Plaisant, C. and Carpendale, S.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {9},
pages = {1520--1536},
title = {{Empirical studies in information visualization: Seven scenarios}},
volume = {18},
year = {2012}
}
@article{Liu2013,
author = {Liu, Y. and Barlowe, S. and Feng, Y. and Yang, J. and Jiang, M.},
journal = {Information Visualization},
number = {1},
pages = {25--43},
title = {{Evaluating exploratory visualization systems: A user study on how clustering-based visualization systems support information seeking from large document collections}},
volume = {12},
year = {2013}
}
@article{Luhn1957,
author = {Luhn, H. P.},
journal = {IBM J. Research and Development},
number = {4},
pages = {309--317},
title = {{A statistical approach to mechanized encoding and searching of literary information}},
volume = {1},
year = {1957}
}
@article{Marchionini2006,
author = {Marchionini, G.},
journal = {Communications of the ACM},
number = {4},
pages = {41},
title = {{Exploratory search: from finding to understanding}},
volume = {49},
year = {2006}
}
@article{Mathison1988,
author = {Mathison, S.},
journal = {Educational Researcher},
number = {2},
pages = {13},
title = {{Why Triangulate?}},
volume = {17},
year = {1988}
}
@article{Mayr2011,
author = {Mayr, E. and Smuc, M. and Risku, H.},
journal = {Information Visualization},
number = {3},
pages = {232--247},
title = {{Many roads lead to Rome: Mapping users' problem-solving strategies}},
volume = {10},
year = {2011}
}
@incollection{McGrath1995,
author = {McGrath, J. E.},
booktitle = {Readings in Human-Computer Interaction: Towards the Year 2000},
edition = {2nd},
editor = {Baecker, R. M. and Grudin, J. and Buxton, W. and Greenberg, S.},
pages = {152--169},
publisher = {Morgan Kaufmann},
title = {{Methodology matters: Doing research in the behavioral and social sciences}},
year = {1995}
}
@article{Milligan1983,
author = {Milligan, G. W. and Soon, S. C. and Sokol, L. M.},
journal = {IEEE Trans. Pattern Analysis and Machine Intelligence (PAMI)},
number = {1},
pages = {40--47},
title = {{The effect of cluster size, dimensionality, and the number of clusters on recovery of true cluster structure}},
year = {1983}
}
@article{Monroe2008,
author = {Monroe, B. L. and Colaresi, M. P. and Quinn, K. M.},
journal = {Political Analysis},
number = {4},
pages = {372--403},
title = {{Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict}},
volume = {16},
year = {2008}
}
@article{Munzner2011a,
author = {Munzner, T and Barsky, A. and Williams, M.},
journal = {Scientific Visualization: Interactions, Features, Metaphors. Dagstuhl Follow-Ups},
number = {17},
pages = {240--259},
title = {{Reflections on QuestVis: A visualization system for an environmental sustainability model}},
volume = {2},
year = {2011}
}
@article{North2006,
author = {North, C.},
journal = {IEEE Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {6--9},
title = {{Toward measuring visualization insight}},
volume = {26},
year = {2006}
}
@article{O'Brien2010,
author = {O'Brien, T. M. and Ritz, A. M. and Raphael, B. J. and Laidlaw, D. H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {918--26},
title = {{Gremlin: an interactive visualization model for analyzing genomic rearrangements.}},
volume = {16},
year = {2010}
}
@inproceedings{Oesterling2011,
author = {\"{O}sterling, P. and Scheuermann, G. and Teresniak, S. and Heyer, G. and Koch, S. and Ertl, T. and Weber, G. H.},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
title = {{Two-stage framework for a topology-based projection and visualization of classified document collections}},
pages = {91--98},
year = {2011}
}
@article{Paulovich2008,
author = {Paulovich, F. V. and Nonato, L. G. and Minghim, R. and Levkowitz, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {564--75},
title = {{Least square projection: a fast high-precision multidimensional projection technique and its application to document mapping.}},
volume = {14},
year = {2008}
}
@article{Perer2009,
author = {Perer, A. and Shneiderman, B.},
journal = {IEEE Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {39--51},
title = {{Integrating statistics and visualization for exploratory power: from long-term case studies to design guidelines.}},
volume = {29},
year = {2009}
}
@book{Pirolli2009,
address = {New York, NY, USA.},
author = {Pirolli, P.},
publisher = {Oxford University Press, Inc.},
title = {{Information Foraging Theory: Adaptive Interaction with Information}},
year = {2009}
}
@inproceedings{Plaisant2004,
author = {Plaisant, C.},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {109},
title = {{The challenge of information visualization evaluation}},
year = {2004}
}
@inproceedings{Pohl2010,
author = {Pohl, M. and Wiltner, S. and Miksch, S.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Exploring information visualization - Describing different interaction patterns}},
year = {2010}
}
@article{Prown1982,
author = {Prown, J. D.},
journal = {Winterthur Portfolio},
number = {1},
pages = {1--19},
title = {{Mind in Matter: An Introduction to Material Culture Theory and Method}},
volume = {17},
year = {1982}
}
@inproceedings{Rester2007a,
author = {Rester, M. and Pohl, M. and Wiltner, S. and Hinum, K. and Miksch, S. and Popow, C. and Ohmann, S.},
booktitle = {Proc. Intl. Conf. Information Visualization},
pages = {693--700},
publisher = {Ieee},
title = {{Evaluating an InfoVis Technique Using Insight Reports}},
year = {2007}
}
@article{Ryan2003,
author = {Ryan, G. W. and Bernard, H. R.},
journal = {Field Methods},
number = {1},
pages = {85--109},
title = {{Techniques to Identify Themes}},
volume = {15},
year = {2003}
}
@inproceedings{Saraiya2010,
author = {Saraiya, P. and North, C. and Duca, K.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Comparing benchmark task and insight evaluation methods on timeseries graph visualizations}},
year = {2010}
}
@inproceedings{Saraiya2004,
author = {Saraiya, P. and North, C. and Duca, K.},
booktitle = {Proc. IEEE Symp. Information Visualization},
pages = {1--8},
publisher = {Ieee},
title = {{An evaluation of microarray visualization tools for biological insight}},
year = {2004}
}
@article{Saraiya2005b,
author = {Saraiya, P. and North, C. and Duca, K.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {443--456},
title = {{An insight-based methodology for evaluating bioinformatics visualizations}},
volume = {11},
year = {2005}
}
@article{Saraiya2006,
author = {Saraiya, P. and North, C. and Lam, V. and Duca, K.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1511--1522},
title = {{An insight-based longitudinal study of visual analytics}},
volume = {12},
year = {2006}
}
@article{Sedlmair2011,
author = {Sedlmair, M. and Isenberg, P. and Baur, D. and Butz, A.},
journal = {Information Visualization},
number = {3},
pages = {248--266},
title = {{Information visualization evaluation in large companies: Challenges, experiences and recommendations}},
volume = {10},
year = {2011}
}
@article{Sedlmair2012b,
author = {Sedlmair, M. and Meyer, M. and Munzner, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2431--2440},
title = {{Design study methodology: Reflections from the trenches and the stacks}},
volume = {18},
year = {2012}
}
@article{Seo2006,
author = {Seo, J. and Shneiderman, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {311--322},
title = {{Knowledge discovery in high-dimensional data: case studies and a user survey for the rank-by-feature framework.}},
volume = {12},
year = {2006}
}
@inproceedings{Shneiderman1996,
author = {Shneiderman, B.},
booktitle = {Proc. IEEE Symp. Visual Languages},
pages = {336--343},
title = {{The eyes have it: A task by data type taxonomy for information visualizations}},
year = {1996}
}
@inproceedings{Shneiderman2006,
author = {Shneiderman, B. and Plaisant, C.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Strategies for evaluating information visualization tools: Multi-dimensional in-depth long-term case studies}},
year = {2006}
}
@book{Silverman2007,
author = {Silverman, D.},
publisher = {Thousand Oaks, CA: Sage},
title = {{A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research}},
year = {2007}
}
@inproceedings{Springmeyer1992,
author = {Springmeyer, R. R. and Blattner, M. M. and Max, N. L.},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {235--242},
title = {{A characterization of the scientific data analysis process}},
year = {1992}
}
@article{Sutcliffe2000,
author = {Sutcliffe, A. G. and Ennis, M. and Hu, J.},
journal = {International J. of Human-Computer Studies (IJHCS)},
number = {5},
pages = {741--763},
title = {{Evaluating the effectiveness of visual user interfaces for information retrieval}},
volume = {53},
year = {2000}
}
@book{Thomas2005,
author = {Thomas, J. J. and Cook, K. A.},
publisher = {IEEE},
title = {{Illuminating the path: The research and development agenda for visual analytics}},
volume = {54},
year = {2005}
}
@article{Toms2000,
author = {Toms, E. G.},
journal = {International J. of Human-Computer Studies (IJHCS)},
number = {3},
pages = {423--452},
title = {{Understanding and facilitating the browsing of electronic text}},
volume = {52},
year = {2000}
}
@unpublished{Torget2010,
author = {Torget, A. J. and Mihalcea, R. and Christensen, J. and Mcghee, G.},
title = {{Mapping texts: Combining text-mining and geo-visualization to unlock the research potential of historical newspapers}},
year = {2010}
}
@inproceedings{Tory2008,
author = {Tory, M. and Staub-French, S.},
booktitle = {Proc. ACM BELIV Workshop},
title = {{Qualitative analysis of visualization: A building design field study}},
year = {2008}
}
@article{Tory2009,
author = {Tory, M. and Swindells, C. and Dreezer, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1033--1039},
title = {{Comparing dot and landscape spatializations for visual memory differences.}},
volume = {15},
year = {2009}
}
@inproceedings{Valiati2006,
author = {Valiati, E. R. A. and Pimenta, M. S. and Freitas, C. M. D. S.},
booktitle = {Proc. ACM BELIV Workshop},
pages = {1--6},
publisher = {ACM},
title = {{A taxonomy of tasks for guiding the evaluation of multidimensional visualizations}},
year = {2006}
}
@inproceedings{VanWijk2005,
author = {van Wijk, J. J.},
booktitle = {Proc. IEEE Symp. Visualization},
pages = {79--86},
title = {{The value of visualization}},
year = {2005}
}
@book{Vicente1999,
author = {Vicente, K. J.},
publisher = {CRC Press},
title = {{Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work}},
year = {1999}
}
@article{Weaver2010a,
author = {Weaver, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {2},
pages = {192--204},
title = {{Cross-filtered views for multidimensional visual analysis}},
volume = {16},
year = {2010}
}
@inproceedings{Winckler2004,
author = {Winckler, M. A. and Palanque, P. and Freitas, C. M. D. S.},
booktitle = {Proc. ACM Intl. Workshop on Task Models and Diagrams for User Interface Design},
title = {{Tasks and scenario-based evaluation of information visualization techniques}},
year = {2004}
}
@inproceedings{Wise1995,
author = {Wise, J. A. and Thomas, J .J. and Pennock, K. and Lantrip, D. and Pottier, M. and Schur, A. and Crow, V.},
booktitle = {Proc. IEEE Symp. Information Visualization},
pages = {51--58},
title = {{Visualizing the non-visual: Spatial analysis and interaction with information from text documents}},
year = {1995},
note = {http://in-spire.pnnl.gov}}
}
@article{Yang2010,
author = {Yang, J. and Luo, D. and Liu, Y.},
journal = {IEEE Computer Graphics and Applications (CG\&A)},
number = {5},
pages = {32--41},
title = {{Newdle: Interactive visual exploration of large online news collections}},
volume = {30},
year = {2010}
}
@inproceedings{Yang2011,
author = {Yang, T. I. and Torget, A. J. and Mihalcea, R.},
booktitle = {Proc. Assoc. Computational Linguistics Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (ACL LATECH)},
pages = {96--104},
title = {{Topic Modeling on Historical Newspapers}},
year = {2011}
}
@article{Yi2007,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, J. A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1224--31},
title = {{Toward a deeper understanding of the role of interaction in information visualization.}},
volume = {13},
year = {2007}
}
@article{Zha2002,
address = {New York, New York, USA},
author = {Zha, H.},
journal = {Proc. ACM Conf. Information Retrieval (SIGIR)},
pages = {113--120},
publisher = {ACM Press},
title = {{Generic summarization and keyphrase extraction using mutual reinforcement principle and sentence clustering}},
year = {2002}
}
@inproceedings{Ziemkiewicz2012a,
author = {Ziemkiewicz, C. and Gomez, S. and Laidlaw, D. H.},
booktitle = {Proc. of the conference on Human factors in computing systems (CHI)},
pages = {1655--1658},
title = {{Analysis within and between graphs: Observed user strategies in immunobiology visualization}},
year = {2012}
}
@article{Stasko2008,
author = {Stasko, J. and Gorg, C. and Spence, R.},
title = {{Jigsaw: supporting investigative analysis through interactive visualization}},
journal = {Information Visualization},
number = {2},
pages = {118--132},
volume = {7},
year = {2008}
}
@article{Havre2002,
author = {Havre, S. and Hetzler, E. and Whitney, P. and Nowell, L.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {1},
pages = {9--20},
title = {{ThemeRiver: Visualizing thematic changes in large document collections}},
volume = {8},
year = {2002}
}
@article{Viegas2009,
author = {Vi\'{e}gas, F. B. and Wattenberg, M. and Feinberg, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1137--1144},
title = {{Participatory visualization with Wordle}},
volume = {15},
year = {2009}
}
@article{Koh2010,
author = {Koh, K. and Lee, B. and Kim, B. and Seo, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1190--1197},
title = {{ManiWordle: Providing flexible control over Wordle}},
volume = {16},
year = {2010}
}
@article{Ericsson1980,
author = {Ericsson, K. A. and Simon, H. A.},
title = {{Verbal reports as data}},
journal = {Psychological Review},
number = {3},
pages = {215--251},
volume = {87},
year = {1980}
}
@inproceedings{Hearst2008,
author = {Hearst, M. A. and Rosner, D.},
title = {{Tag clouds: Data analysis tool or social signaller?}},
booktitle = {Proc. Hawaii Intl. Conf. System Sciences (HICSS)},
year = {2008}
}
@article{Tory2007,
author = {Tory, M. and Sprague, D. and Wu, F. and So, W. Y. and Munzner, T.},
title = {{Spatialization design: comparing points and landscapes.}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1262--1269},
volume = {13},
year = {2007}
}
@article{Sedlmair2012c,
author = {Sedlmair, M. and Frank, A. and Munzner, T. and Butz, A.},
title = {{RelEx: Visualization for actively changing overlay network specifications}},
journal ={IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
year = {2012},
volume = {18},
number = {12},
pages = {2729--2738}
}
@inproceedings{Lohmann2009,
author = {Lohmann, S. and Ziegler, J. and Tetzlaff, L.},
title = {{Comparison of tag cloud layouts: Task-related performance and visual exploration}},
booktitle = {Proc. IFIP INTERACT},
pages = {392--404},
year = {2009}
}
@inproceedings{Cui2010,
author = {Cui, W. and Wu, Y. and Liu, S. and Wei, F. and Zhou, M. X. and Qu, H.},
title = {{Context preserving dynamic word cloud visualization}},
booktitle = {Proc. IEEE Symp. Pacific Visualization},
pages = {121--128},
year = {2010}
}
@inproceedings{Rivadeneira2007,
author = {Rivadeneira, A. W. and Gruen, D. M. and Muller, M. J. and Millen, D. R.},
title = {{Getting our head in the clouds: Toward evaluation studies of tagclouds}},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {995},
year = {2007}
}
@inproceedings{Schrammel2009,
author = {Schrammel, J. and Leitner, M. and Tscheligi, M.},
title = {{Semantically structured tag clouds: An empirical evaluation of clustered presentation approaches}},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
pages = {2037},
year = {2009}
}
@inproceedings{Ho2012,
author = {Ho, Q. and Eisenstein, J. and Xing, E. P.},
title = {{Document hierarchies from text and links}},
booktitle = {Proc. ACM Conf. World Wide Web (WWW)},
pages = {739--748},
year = {2012}
}
@article{Kellar2007,
author = {Kellar, M. and Watters, C. and Shepherd, M.},
title = {{A goal-based classification of web information tasks}},
journal = {Proc. American Soc. Information Science and Technology},
number = {1},
pages = {1--22},
volume = {43},
year = {2007}
}
@article{Chen2002,
author = {Chen, H. and Schroeder, J. and Huack, R. V. and Ridgeway, L. and Atabachsh, H. and Gupta, H. and Boarman, C. and Rasmussen, K. and Clements, A. W.},
title = {{COPLINK connect: information and knowledge management for law enforcment}},
journal = {Decision Support Systems},
pages = {271--285},
volume = {34},
year = {2002}
}
@inproceedings{Jonker2005,
author = {Jonker, D. and Wright, W. and Schroh, D. and Proulx, P. and Cort, B.},
title = {{Information Triage with TRIST}},
booktitle = {Proc. Conf. Intelligence Analysis},
pages = {1--6},
year = {2005}
}
@article{Chen2003,
author = {Chen, H. and Zeng, D. and Atabakhsh, H. and Wyzga, W. and Schroeder, J.},
title = {{COPLINK: Managing law enforcement data and knowledge}},
journal = {Communications of the ACM},
number = {1},
volume = {46},
year = {2003}
}
@article{Lloyd2011,
author = {Lloyd, D. and Dykes, J.},
title = {{Human-centered approaches in geovisualization design: investigating multiple methods through a long-term case study.}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2498--2507},
volume = {17},
year = {2011}
}
@article{Munzner2009,
author = {Munzner, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {921--928},
title = {{A nested model for visualization design and validation}},
volume = {15},
year = {2009}
}
@inproceedings{Valiati2008,
author = {Valiati, E. R. A. and Freitas, C. M. D. S. and Pimenta, M. S.},
title = {{Using Multi-dimensional In-depth Long-term Case Studies for information visualization evaluation}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2008}
}
@phdthesis{Ingram2013,
author = {Ingram, S.},
school = {University of British Columbia},
title = {{Practical Guidance for Dimensionality Reduction: User Guidance, Costly Distances, and Document Data}},
year = {2013},
type = {{PhD} dissertation}
}
@article{Cunningham2011,
title = {{Challenges in Document Mining (Dagstuhl Seminar 11171)}},
pages = {65--99},
journal = {Dagstuhl Reports},
year = {2011},
volume = {1},
number = {4},
author = {Cunningham, H. and Fuhr, N. and Stein, B. M.},
publisher = {Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik}
}
@inproceedings{Weaver2007,
author = {Weaver, C.},
title = {{Patterns of coordination in Improvise visualizations}},
booktitle = {Proc. SPIE--IS\&T Visualization and Data Analysis (VDA)},
year = {2007}
}
@inproceedings{Ingram2010,
author = {Ingram, S. and Munzner, T. and Irvine, V. and Tory, M. and Bergner, S. and M\"{o}ller, T.},
title = {{DimStiller: Workflows for dimensional analysis and reduction}},
booktitle = {Proc. IEEE Symp. Visual Analytics Software and Technologies (VAST)},
pages = {3--10},
year = {2010}
}
@article{Salton1988,
author = {Salton, G. and Buckley, C.},
title = {{Term-weighting approaches in automatic text retrieval}},
pages = {513--523},
journal = {Information Processing \& Management},
year = {1988},
volume = {24},
number = {5}
}
@inproceedings{Meyer2012,
author = {Meyer, M. and Sedlmair, M. and Munzner, T.},
title = {{The four-level nested model revisited: blocks and guidelines}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2012}
}
@article{Lam2010,
author = {Lam, H. and Munzner, T.},
journal = {Synthesis Lectures on Visualization},
number = {1},
pages = {1-117},
publisher = {Morgan and Claypool},
title = {{A guide to visual multi-level interface design from synthesis of empirical study evidence}},
volume = {1},
year = {2010}
}
@inproceedings{Lee2006,
author = {Lee, B. and Plaisant, C. and Parr, C. S. and Fekete, J. D. and Henry, N.},
title = {{Task taxonomy for graph visualization}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2006}
}
@article{Kang2012,
author = {Kang, Y. A. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {2869--2878},
title = {{Examining the use of a visual analytics system for sensemaking tasks: Case studies with domain experts}},
volume = {18},
year = {2012}
}
@inproceedings{Kang2009,
author = {Kang, Y. A. and Gorg, C. and Stasko, J.},
title = {{Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology},
doi = {10.1109/VAST.2009.5333878},
pages = {139--146},
year = {2009}
}
@article{Gorg2013,
author = {G\"{o}rg, C. and Liu, Z. and Kihm, J. and Choo, J. and Park, H. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {10},
pages = {1646--1663},
title = {{Combining computational analyses and interactive visualization for document exploration and sensemaking in Jigsaw}},
volume = {19},
year = {2013}
}
@article{Dou2013,
author = {Dou, W. and Yu, L. and Wang, X. and Ma, Z. and Ribarsky, W.},
number = {12},
pages = {2002--2011},
title = {{HierarchicalTopics: Visually exploring large text collections using topic hierarchies}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
volume = {19},
year = {2013}
}
@book{Schwaiger2003,
editor = {Schwaiger, M. and Opitz, O.},
publisher = {Springer},
series = {Studies in Classification, Data Analysis, and Knowledge Organization},
title = {{Exploratory Data Analysis in Empirical Research}},
year = {2003}
}
@book{Kiers2000,
editor = {Kiers, H. A. L. and Rasson, J. P. and Groenen, P. J. F. and Schader, M.},
publisher = {Springer},
series = {Studies in Classification, Data Analysis, and Knowledge Organization},
title = {{Data Analysis, Classification, and Related Methods}},
year = {2000}
}
@article{Tversky1977,
author = {Tversky, A.},
journal = {Psychological Review},
number = {4},
pages = {327--352},
title = {{Features of similarity}},
volume = {84},
year = {1977}
}
@inproceedings{Hatzivassiloglou2000,
author = {Hatzivassiloglou, V. and Gravano, L. and Maganti, A.},
booktitle = {Proc. ACM SIGIR Intl. Conf. Research and Development in Information Retrieval},
pages = {224--231},
title = {{An investigation of linguistic features and clustering algorithms for topical document clustering}},
year = {2000}
}
@article{Grossman2011,
author = {Grossman, M. R. and Cormack, G. V.},
journal = {Richmond Journal of Law and Technology},
number = {3},
pages = {1--33},
title = {{Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review}},
volume = {XVII},
year = {2011}
}
@incollection{Holtzblatt1993,
author = {Holtzblatt, K. and Jones, S.},
title = {{Contextual inquiry: A participatory technique for system design}},
booktitle = {Participatory Design: Principles and Practices},
chapter = {9},
editor = {Schuler, D. and Namioka, A.},
pages = {177--210},
publisher = {Lawrence Erlbaum Associates},
year = {1993}
}
@article{Kang2010,
author = {Kang, Y. A. and G笠C. and Stasko, J. T.},
title = {{How can visual analytics assist investigative analysis? Design implications from an evaluation}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {570--583},
volume = {17},
year = {2010}
}
@article{Yang2013,
author = {Yang, J. and Liu, Y. and Zhang, X. and Yuan, X. and Zhao, Y. and Barlowe, S. and Liu, Shixia},
title = {{PIWI: visually exploring graphs based on their community structure}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1034--47},
volume = {19},
year = {2013}
}
@book{Schwartz2003,
author = "Barry Schwartz",
title = "The Paradox of Choice: Why More Is Less",
publisher = "Ecco",
year = 2003
}
@inproceedings{Nielsen1993,
author = {Nielsen, J. and Landauer, T. K.},
title = {A mathematical model of the finding of usability problems},
booktitle = {Proc. ACM Conf. Human Factors in Computing Systems (CHI)},
year = {1993},
pages = {206--213}
}
@article{Pham2005,
author = {Pham, D. and Dimov, S. and Nguyen, C.},
title = {{Selection of K in K-means clustering}},
journal = {J. Mechanical Engineering Science (Proc. IMechE)},
volume = {219},
issue = {1},
pages = {103--119},
year = {2005}
}
@article{Wainer2007,
author = {Wainer, H and Brown, L. M.},
journal = {Handbook of Statistics},
title = {{Three statistical paradoxes in the interpretation of group differences: Illustrated with medical school admission and licensing data}},
volume = {26},
year = {2007}
}
@article{Liu2012,
author = {Liu, S. and Zhou, M. X. and Pan, S. and Song, Y. and Qian, W. and Cai, W. and Lian, X.},
journal = {ACM Trans. Intelligent Systems and Technology},
number = {2},
title = {{TIARA: Interactive, topic-based visual text summarization and analysis}},
volume = {3},
year = {2012}
}
@inproceedings{Hatzivassiloglou2001,
author = {Hatzivassiloglou, V. and Klavans, J. L. and Holcombe, M. L. and Barzilay, R. and Kan, M. Y. and McKeown, K. R.},
title = {{Simfinder: A flexible clustering tool for summarization}},
booktitle = {Procl. NAACL Workshop on Automatic Summarization},
year = {2001}
}
@inproceedings{Franz2001,
author = {Franz, M. and McCarley, J. S. and Ward, T. and Zhu, W. J.},
title = {{Unsupervised and supervised clustering for topic tracking}},
booktitle = {Proc. ACM SIGIR Intl. Conf. Research and Development in Information Retrieval},
pages = {310--317},
year = {2001}
}
@inproceedings{Paulovich2007,
author = {Paulovich, F. V. and Oliveira, M. C. F. and Minghim, R.},
title = {The projection explorer: A flexible tool for projection-based multidimensional visualization},
booktitle = {Proc. IEEE Brazilian Symp. Computer Graphics and Image Processing},
year = {2007},
pages = {27--36}
}
@inproceedings{Granitzer2004,
title={Evaluating a system for interactive exploration of large, hierarchically structured document repositories},
author={Granitzer, M. and Kienreich, W. and Sabol, V. and Andrews, K. and Klieber, W.},
booktitle={Proc. IEEE Symp. Information Visualization (InfoVis)},
pages={127--134},
year={2004}
}
@article{Kandel2011,
author = {Kandel, S. and Heer, J. and Plaisant, C. and Kennedy, J. and van Ham, F. and Riche, N. H. and Weaver, C. and Lee, B. and Brodbeck, D. and Buono, P.},
title = {{Research directions in data wrangling: Visualizations and transformations for usable and credible data}},
journal = {Information Visualization},
number = {4},
pages = {271--288},
volume = {10},
year = {2011}
}
% 5 adoption papers from Lam 2012
@article{Gotz2009,
author = {Gotz, D. and Zhou, M. X.},
title = {{Characterizing users橳ual analytic activity for insight provenance}},
journal = {Information Visualization},
number = {1},
pages = {42--55},
volume = {8},
year = {2009}
}
@article{Wang2007,
author = {Wang, W. and Lu, A.},
title = {{Interactive wormhole detection and evaluation}},
journal = {Information Visualization},
number = {1},
pages = {3--17},
volume = {6},
year = {2007}
}
@article{Kincaid2005,
author = {Kincaid, R. and Ben-Dor, A. and Yakhini, Z.},
title = {{Exploratory visualization of array-based comparative genomic hybridization}},
journal = {Information Visualization},
number = {3},
pages = {176--190},
volume = {4},
year = {2005}
}
@article{McKeon2009,
author = {McKeon, M.},
title = {{Harnessing the web information ecosystem with wiki-based visualization dashboards}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1081--1088},
volume = {15},
year = {2009}
}
@inproceedings{Robinson2008,
author = {Robinson, A. C.},
title = {{Collaborative synthesis of visual analytic results}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {67--74},
year = {2008}
}
@article{Meyer2014,
author = {Meyer, M. and Sedlmair, M. and Quinan, P. S. and Munzner, T.},
title = {{The nested blocks and guidelines model}},
journal = {Information Visualization},
note = {doi.org/10.1177/1473871613510429},
volume = {In press}
}
@inproceedings{Chaney2012,
title = {Visualizing topic models},
author = {Chaney, A. J.-B. and Blei, D. M.},
booktitle = {Proc. Intl. AAAI Conf. Weblogs and Social Media (ICWSM)},
pages = {419--422},
year = {2012}
}
@article{Cao2010,
author={Cao, N. and Sun, J. and Lin, L-R. and Gotz, D. and Liu, S. and Qu, H.},
title = {FacetAtlas: Multifaceted visualization for rich text corpora},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
year = {2010},
volume = {16},
number = {6},
pages = {1172--1181}
}
@inproceedings{Lee2009,
author = {Lee, B. and Smith, G. and Robertson, G. G. and Czerwinski, M. and Tan, D. S},
title = {{FacetLens: Exposing trends and relationships to support sensemaking within faceted datasets}},
booktitle = {Proc. ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
pages = {1293--1302},
year = {2009}
}
@inproceedings{Iwata2008,
author = {Iwata, T. and Yamada, T. and Ueda, N.},
title = {{Probabilistic latent semantic visualization: Topic model for visualizing documents}},
booktitle = {Proc. ACM SIGKDD Intl. Conf. Knowledge Discovery and Data Mining},
pages = {363--371},
year = {2008}
}
@article{Correll2011,
author = {Correll, M. and Witmore, M. and Gleicher, M.},
title = {{Exploring collections of tagged text for literary scholarship}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {731--740},
volume = {30},
year = {2011}
}
@incollection{Seifert2014,
author = {Seifert, C. and Sabol, V. and Kienreich, W. and Lex, E. and Granitzer, M.},
booktitle = {Large Scale Data Analytics},
title = {{Visual analysis and knowledge discovery for text}},
editor = {Gkoulalas-Divanis, A. and Labbi, A.},
pages = {189--218},
publisher = {Springer},
year = {2014}
}
@article{Gretarsson2012,
author = {Gretarsson, B. and Oﮯvan, J. and Bostandjiev, S. and H\"{o}llerer, T. and Asuncion, A. and Newman, D. and Smyth, P.},
title = {{TopicNets: Visual analysis of large test corpora with topic modeling}},
journal = {ACM Trans. Intelligent Systems and Technology},
number = {2},
pages = {1--26},
volume = {3},
year = {2012}
}
@article{Strobelt2009,
author = {Strobelt, H. and Oelke, D. and Rohrdantz, C. and Stoffel, A. and Keim, D. A. and Deussen, O.},
title = {{Document cards: a top trumps visualization for documents}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1145--1152},
volume = {15},
year = {2009}
}
@article{Smith2006,
author = {Smith, G. and Czerwinski, M. and Meyers, B. and Robbins, D. and Robertson, G. and Tan, D. S.},
title = {{FacetMap: A scalable search and browse visualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {797--804},
volume = {12},
year = {2006}
}
@inproceedings{Miller1998,
author = {Miller, N. E. and Wong, P. C. and Brewster, M. and Foote, H.},
title = {{Topic Islands: A wavelet-based text visualization system}},
booktitle = {Proc. IEEE Visualization},
pages = {189--196},
volume = {98},
year = {1998}
}
@article{Ham2009,
author = {van Ham, F. and Perer, A.},
title = {{"Search, show context, expand on demand": supporting large graph exploration with degree-of-interest}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {953--960},
volume = {15},
year = {2009}
}
@article{Gregor2013,
author = {Gregor, S. and Hevner, A. R.},
title = {{Positioning and presenting design science research for maximum impact}},
journal = {MIS Quartlery},
number = {2},
volume = {37},
year = {2013}
}
@article{Hevner2004,
author = {Hevner, A. R. and Ram, S. and March, S. T.},
title = {{Design science in information systems research}},
journal = {MIS Quartlery},
number = {1},
pages = {75--105},
volume = {28},
year = {2004}
}
@article{Cui2011
author = {Cui, W. and Liu, S. and Tan, L. and Shi C, and Song, Y. and Gao, Z. J. and Tong, X. and Qu, H.},
title = {{TextFlow: Towards Better Understanding of Evolving Topics in Text}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2412--2421},
volume = {17},
year = {2011}
}
@misc{ISO,
author = {ISO},
title = {{ISO 13407:1999 Human-centred design processes for interactive systems}},
year = {1999}
}
@article{Gorg2013c,
author = {Gorg, C. and Liu, Z. and Stasko, J.},
doi = {10.1177/1473871613495674},
journal = {Information Visualization},
pages = {1--11},
title = {{Reflections on the evolution of the Jigsaw visual analytics system}},
volume = {(In press)},
year = {2013}
}
@article{Kang2012a,
author = {Kang, Y.-A. and Stasko, J.},
journal = {Information Visualization},
number = {2},
pages = {134--158},
title = {{Characterizing the intelligence analysis process through a longitudinal field study: Implications for visual analytics}},
volume = {13},
year = {2012}
}
@article{Kwon2012,
author = {Kwon, B. C. and Javed, W. and Ghani, S. and Elmqvist, N. and Yi, J. S. and Ebert, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {11},
pages = {1992--2004},
title = {{Evaluating the role of time in investigative analysis of document collections}},
volume = {18},
year = {2012}
}
@inproceedings{Smith2014,
author = {Smith, A. and Hawes, T. and Myers, M.},
title = {{HiᲣhie: Interactive visualization for hierarchical topic models}},
booktitle = {Proc. ACL Workshop on Interactive Language Learning, Visualization, and Interfaces},
pages = {1--6},
year = {2014}
}
@inproceedings{Gardner2010,
author = {Gardner, M. J. and Lutes, J. and Lund, F. and Hansen, J. and Walker, D. and Ringger, E. and Seppi, K.},
title = {{The topic browser: An interactive tool for browsing topic models}},
booktitle = {Proc. NIPS Workshop on Challenges of Data Visualization},
year = {2010}
}
@incollection{Fabrikant2005,
author = {Fabrikant, Sara Irina and Santa, California},
title = {{Cognitively plausible information visualization}},
booktitle = {Exploring Geovisualization},
chapter = {35},
editor = {Dykes, J. and MacEachren, A. M. and Kraak, M.
pages = {667--690},
year = {2005}
}
@inproceedings{Chuang2012,
author = {Chuang, J. and Manning, C. D. and Heer, J.},
title = {{Termite: visualization techniques for assessing textual topic models}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {74--77},
year = {2012}
}
@article{Paulovich2008a,
author = {Paulovich, F. V. and Minghim, R.},
title = {{HiPP: A novel hierarchical point placement strategy and its application to the exploration of document collections}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1229--1236},
volume = {14},
year = {2008}
}
}}}
{{{
@book{Ware2004,
address = {San Fransisco, CA, USA},
author = {Ware, C.},
edition = {2},
pages = {228--237},
publisher = {Morgan Kaufmann},
title = {{Information Visualization: Perception for Design}},
year = {2004}
}
@article{Ware1995,
author = {Ware, C. and Knight, W.},
doi = {10.1145/200972.200974},
journal = {ACM Trans. Graphics (TOG)},
number = {1},
pages = {3--20},
title = {{Using visual texture for information display}},
url = {http://portal.acm.org/citation.cfm?doid=200972.200974},
volume = {14},
year = {1995}
}
@book{Williams,
author = {Williams},
pages = {11--20, 226--230, 273--284},
title = {{Animator's Survival Kit}}
}
@article{Wolfe2004,
author = {Wolfe, J. M. and Horowitz, T. S.},
doi = {10.1038/nrn1411},
journal = {Nature Reviews: Neuroscience},
number = {6},
pages = {495--501},
title = {{What attributes guide the deployment of visual attention and how do they do it?}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/15152199},
volume = {5},
year = {2004}
}
@article{Wood2006,
author = {Wood, S. and Cox, R. and Cheng, P.},
doi = {10.1016/j.chb.2005.12.007},
journal = {Computers in Human Behavior},
number = {4},
pages = {588--602},
title = {{Attention design: Eight issues to consider}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0747563205001172},
volume = {22},
year = {2006}
}
@incollection{Zacks2003,
address = {New York, NY, USA},
author = {Zacks, J. M. and Magliano, J. P.},
title = {{Film, Narrative, and Cognitive Neuroscience}},
booktitle = {Art and the Senses},
editor = {Melcher, D. P. and Bacci, F.},
pages = {1--20},
publisher = {Oxford University Press},
volume = {1},
year = {2003}
}
@article{Zacks2007,
author = {Zacks, J. M. and Swallow, K. M.},
doi = {10.1111/j.1467-8721.2007.00480.x},
journal = {Current Directions in Psychological Science},
number = {2},
pages = {80--84},
title = {{Event Segmentation}},
url = {http://cdp.sagepub.com/lookup/doi/10.1111/j.1467-8721.2007.00480.x},
volume = {16},
year = {2007}
}
@article{Zacks2001,
author = {Zacks, J. M. and Tversky, B.},
journal = {Psychological Bulletin},
number = {1},
pages = {3--21},
title = {{Event Structure in Perception and Conception}},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.89.6156\&rep=rep1\&type=pdf},
volume = {127},
year = {2001}
}
@article{Tversky2002,
author = {Tversky, B. and {Bauer Morrison}, J. and Betrancourt, M.},
doi = {10.1006/ijhc.1017},
journal = {Intl. J. Human-Computer Studies},
number = {4},
pages = {247--262},
title = {{Animation: can it facilitate?}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581902910177},
volume = {57},
year = {2002}
}
@article{Varakin2004,
author = {Varakin, D. A. and Levin, D. T. and Fidler, R.},
journal = {Human-Computer Interaction},
number = {4},
pages = {389--422},
publisher = {L. Erlbaum Associates Inc.},
title = {{Unseen and unaware: Implications of recent research on failures of visual awareness for human-computer interface design}},
url = {http://portal.acm.org/citation.cfm?id=1466559.1466568},
volume = {19},
year = {2004}
}
@book{Tufte1990,
address = {Cheshire, CT, USA},
author = {Tufte, E. R},
publisher = {Graphics Press},
title = {{Envisioning Information}},
year = {1990}
}
@incollection{Tufte2001,
author = {Tufte, E. R},
title = {{Data-Ink and Graphical Redesign}},
booktitle = {The Visual Display of Quantitative Information},
chapter = {4},
edition = {2},
pages = {91--106},
year = {2001}
}
@incollection{Tufte2001a,
author = {Tufte, E. R},
title = {{Chartjunk: Vibrations, Grids, and Ducks}},
booktitle = {The Visual Display of Quantitative Information},
chapter = {5},
edition = {2},
pages = {107--122},
year = {2001}
}
@incollection{Tufte2001b,
author = {Tufte, E. R},
title = {{Data-Ink Maximization and Graphical Design}},
booktitle = {The Visual Display of Quantitative Information},
chapter = {6},
edition = {2},
pages = {123--137},
year = {2001}
}
@incollection{Tversky2001,
author = {Tversky, B.},
title = {{Spatial schemas in depictions}},
booktitle = {Spatial schemas and abstract thought},
chapter = {4},
pages = {79--111},
publisher = {MIT Press},
url = {http://books.google.com/books?hl=en\&lr=\&id=EbKmYYTf9gMC\&oi=fnd\&pg=PA79\&dq=Spatial+Schemas+in+Depictions\&ots=oN0Ps-McGy\&sig=J68fLrJzJjTYpoAfEFLWLEqCyAs},
year = {2001}
}
@book{McCloudc,
author = {McCloud, S.},
title = {{Understanding Comics}},
pages = {212--235},
publisher = {Kepustakaan Populer Gramedia},
url = {http://scholar.google.com/scholar?hl=en\&btnG=Search\&q=intitle:Understanding+Comics\#0}
}
@article{Merikle1998,
author = {Merikle, P.M. and Daneman, M.},
journal = {Consciousness Studies},
number = {1},
pages = {5--18},
publisher = {Imprint Academic},
title = {{Psychological Investigations of Unconscious Perception}},
url = {http://www.ingentaconnect.com/content/imp/jcs/1998/00000005/00000001/814},
volume = {5},
year = {1998}
}
@book{Monmonier,
author = {Monmonier, M.},
booktitle = {How to Lie With Maps},
chapter = {1-3},
pages = {1--42},
title = {{How to Lie With Maps}}
}
@inproceedings{Muller2003,
author = {M⬠W. and Schumann, H.},
title = {{Visualization Methods for Time-Dependent Data - An Overview}},
booktitle = {Winter Simualtion Conference},
editor = {Chick, S. and Sanchez, P. J. and Ferrin, D. and Morrice, D. J.},
pages = {737--745},
year = {2003}
}
@book{Mullet1995,
author = {Mullet, K. and Sano, D.},
booktitle = {Designing visual interfaces: Communication oriented techniques},
pages = {1--49},
publisher = {SunSoft Press Englewood Cliffs, NJ},
title = {{Designing visual interfaces: Communication oriented techniques}},
url = {http://www.citeulike.org/user/ryanshaw/article/260093},
year = {1995}
}
@incollection{Munzner2009,
author = {Munzner, T.},
title = {{Visualization}},
booktitle = {Fundamentals of Graphics},
chapter = {27},
edition = {3},
editor = {Shirley, Peter},
pages = {675--707},
publisher = {AK Peters},
year = {2009}
}
@article{Murphy1993,
author = {Murphy, S. T. and Zajonc, R. B.},
journal = {Personality and Social Psychology},
keywords = {Adult,Affect,Attention,Awareness,Cognition,Female,Humans,Male,Mental Recall,Set (Psychology)},
number = {5},
pages = {723--39},
title = {{Affect, cognition, and awareness: affective priming with optimal and suboptimal stimulus exposures}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/8505704},
volume = {64},
year = {1993}
}
@article{Newell2004,
author = {Newell, F. N. and Wallraven, C. and Huber, S.},
doi = {10.1167/4.2.5},
journal = {Vision},
number = {2},
pages = {118--129},
publisher = {Association for Research in Vision and Ophthalmology},
title = {{The role of characteristic motion in object categorization}},
url = {http://jwww.journalofvision.org/content/4/2/5.full},
volume = {4},
year = {2004}
}
@inproceedings{Nowell2002,
author = {Nowell, L. and Schulman, R. and Hix, D.},
title = {{Graphical encoding for information visualization: an empirical study}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2002.1173146},
pages = {43--50},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173146},
volume = {2002},
year = {2002}
}
@incollection{Oliva2005,
author = {Oliva, A.},
title = {{Gist of the Scene}},
booktitle = {Neurobiology of Attention},
chapter = {41},
pages = {251--257},
year = {2005}
}
@article{Openshaw1984,
author = {Openshaw, S.},
journal = {Concepts and Techniques in Modern Geography},
title = {{The Modifiable Aereal Unit Problem}},
volume = {38},
year = {1984}
}
@incollection{Palmer11.2.41999,
author = {Palmer, S. E.},
title = {{Selective Attention to Properties}},
booktitle = {Vision Science: Photons to Phenomenology},
chapter = {11.2.4},
pages = {549--554},
publisher = {MIT Press},
year = {1999}
}
@incollection{Palmer3.311999,
author = {Palmer, S. E.},
title = {{Lightness Constancy}},
booktitle = {Vision Science: Photons to Phenomenology},
chapter = {3.31},
pages = {125--133},
publisher = {MIT Press},
year = {1999}
}
@incollection{Palmer61999,
author = {Palmer, S. E.},
title = {{Organizing Objects and Scenes}},
booktitle = {Vision Science: Photons to Phenomenology},
chapter = {6},
pages = {254--309},
publisher = {MIT Press},
year = {1999}
}
@incollection{Palmer31999,
author = {Palmer, S. E.},
title = {{Color Vision}},
booktitle = {Vision Science: Photons to Phenomenology},
chapter = {3},
pages = {94--142},
publisher = {MIT Press},
year = {1999}
}
@inproceedings{Po2003,
author = {Po, B. and Fisher, B. and Booth, K.},
title = {{Pointing and visual feedback for spatial interaction in large-screen display environments}},
booktitle = {Smart Graphics},
pages = {183--196},
publisher = {Springer},
url = {http://www.springerlink.com/index/NNK4TK5GC8VDP96L.pdf},
year = {2003}
}
@article{Rensink2000,
author = {Rensink, R. A.},
doi = {10.1080/135062800394667},
issn = {1350-6285},
journal = {Visual Cognition},
number = {1},
pages = {17--42},
title = {{The Dynamic Representation of Scenes}},
url = {http://www.informaworld.com/openurl?genre=article\&doi=10.1080/135062800394667\&magic=crossref||D404A21C5BB053405B1A640AFFD44AE3},
volume = {7},
year = {2000}
}
@article{Rensink2002,
address = {New York, New York, USA},
author = {Rensink, R. A.},
doi = {10.1145/569005.569015},
journal = {Proc. ACM Intl. Symp. Smart Graphics (SMARTGRAPH)},
pages = {63--70},
title = {{Internal vs. external information in visual perception}},
url = {http://portal.acm.org/citation.cfm?doid=569005.569015},
year = {2002}
}
@article{Rensink2010,
author = {Rensink, R. A. and Baldridge, G.},
title = {{The Perception of Correlation in Scatterplots}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {1203--1210},
url = {http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8659.2009.01694.x/full},
volume = {29},
year = {2010}
}
@article{Rensink1998,
author = {Rensink, R. A. and Enns, J. T.},
journal = {Vision Research},
number = {15-16},
pages = {2489--505},
title = {{Early completion of occluded objects.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/9798011},
volume = {38},
year = {1998}
}
@incollection{Rensink2011,
author = {Rensink, R.A.},
title = {{The management of visual attention in graphic displays}},
booktitle = {Human Attention in Digital Environments},
chapter = {3},
editor = {Roda, C.},
pages = {63--92},
publisher = {Cambridge: University Press},
url = {http://scholar.google.com/scholar?hl=en\&btnG=Search\&q=intitle:The+management+of+visual+attention+in+graphic+displays\#0},
year = {2011}
}
@article{Rheingans2000,
author = {Rheingans, P. L.},
doi = {10.1117/12.384882},
issn = {0277786X},
journal = {Proc. SPIE},
pages = {35--43},
publisher = {Spie},
title = {{Task-based color scale design}},
url = {http://link.aip.org/link/?PSI/3905/35/1\&Agg=doi},
year = {2000}
}
@article{Rohrer1999,
author = {Rohrer, R. M. and Sibert, J. L. and Ebert, D. S.},
doi = {10.1109/38.788797},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {5},
pages = {40--46},
title = {{A shape-based visual interface for text retrieval}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=788797},
volume = {19},
year = {1999}
}
@inproceedings{Rosenholtz2005,
author = {Rosenholtz, R. and Li, Y. and Jin, Z. and Mansfield, J.},
title = {{Feature congestion: A measure of visual clutter}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1167/6.6.827},
number = {6},
pages = {827--827},
url = {http://www.journalofvision.org/lookup/doi/10.1167/6.6.827},
volume = {6},
year = {2005}
}
@incollection{Sekuler1988,
author = {Sekuler, R. and Watamaniuk, S. N. J. and Blake, R.},
title = {{Motion Perception}},
chapter = {4},
number = {1951},
pages = {121--176},
year = {1988}
}
@article{Spence1999,
author = {Spence, R.},
doi = {10.1006/ijhc.1999.0265},
journal = {Intl. J. Human-Computer Studies},
=number = {5},
pages = {919--945},
title = {{A Framework for Navigation}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581999902653},
volume = {51},
year = {1999}
}
@inproceedings{Tang2010,
address = {Savannah, Georgia, USA},
author = {Tang, A. and Pahud, M. and Inkpen, K. M. and Benko, H. and Tang, J. C. and Buxton, W.},
title = {{Three's Company: Understanding Communication Channels in Three-way Distributed Collaboration}},
booktitle = {Proc. ACM Conf. Computer-Supported Collaborative Work (CSCW)},
year = {2010}
}
@incollection{Massironi2001a,
author = {Massironi, M.},
title = {{Ambiguity and Information}},
booktitle = {The Psychology of Graphic Images},
chapter = {9},
pages = {260--266},
publisher = {Psychology Press},
year = {2001}
}
@book{MacEachren1995,
author = {MacEachren, A. M.},
chapter = {3},
number = {13},
pages = {80--101},
publisher = {The Guilford Press},
title = {{How Maps Work: Representation, Visualization, and Design}},
volume = {81},
year = {1995}
}
@article{MacEachren2000,
author = {MacEachren, A. M. and Xiping, D. and Hardisty, F. and Lengerich, G.},
title = {{Exploring high-D spaces with multiform matrices and small multiples}},
doi = {10.1109/INFVIS.2003.1249006},
isbn = {0-7803-8154-8},
journal = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {31--38},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1249006},
year = {2000}
}
@incollection{Massironi2001,
author = {Massironi, M.},
title = {{What Makes a Graphic Image Work}},
booktitle = {The Psychology of Graphic Images},
chapter = {5},
doi = {10.1016/S0072-9752(08)02151-9},
pages = {97--129},
publisher = {Psychology Press},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20123464},
volume = {95},
year = {2001}
}
@incollection{Adelson2000,
author = {Adelson, E. H.},
title = {{Lightness Perception and Lightness Illusions}},
booktitle = {The New Cognitive Neurosciences},
chapter = {24},
edition = {2nd},
editor = {Gazzaniga, M.},
pages = {339--351},
publisher = {MIT Press},
url = {http://books.google.com/books?hl=en\&lr=\&id=0wx17lC075EC\&oi=fnd\&pg=PA339\&dq=Lightness+Perception+and+Lightness+Illusions\&ots=RlKNCuF-OA\&sig=4jZCqXN3uOg9eQdaQCesimHA-zY},
year = {2000}
}
@article{Aginsky1997,
author = {Aginsky, V. and Harris, C. and Rensink, R. A. and Beusmans, J.},
journal = {Environmental Psychology},
pages = {317--331},
title = {{Two Strategies for Learning a Route in a Driving Simulator}},
volume = {17},
year = {1997}
}
@inproceedings{Baecker1991,
author = {Baecker, R. M. and Small, I. and Mander, R.},
title = {{Bringing Icons to Life}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
number = {2},
pages = {444--449},
volume = {68},
year = {1991}
}
@article{Bar2004,
author = {Bar, M.},
doi = {10.1038/nrn1476},
journal = {Nature Reviews: Neuroscience},
number = {8},
pages = {617--29},
title = {{Visual objects in context}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/15263892},
volume = {5},
year = {2004}
}
@book{Bardel2001,
author = {Bardel, W. H.},
title = {{Depth Cues For Information Design}},
pages = {1--35},
year = {2001}
}
@inproceedings{Bartram1997,
author = {Bartram, L.},
title = {{Perceptual and interpretative properties of motion for information visualization}},
booktitle = {Proc. ACM Workshop. New Paradigms in Information Visualization and Manipulation (NPIV)},
doi = {10.1145/275519.275520},
pages = {3--7},
url = {http://portal.acm.org/citation.cfm?doid=275519.275520},
year = {1997}
}
@inproceedings{Baudisch2002,
author = {Baudisch, P. and Good, N. and Bellotti, V. and Schraedley, P.},
title = {{Keeping things in context: a comparative evaluation of focus plus context screens, overviews, and zooming}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
number = {4},
pages = {259--266},
url = {http://portal.acm.org/citation.cfm?id=503376.503423},
year = {2002}
}
@inproceedings{Baudisch2001,
author = {Baudisch, P. and Good, N. and Stewart, P.},
title = {{Focus plus context screens: combining display technology with visualization techniques}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
pages = {31--40},
url = {http://portal.acm.org/citation.cfm?id=502354},
volume = {3},
year = {2001}
}
@incollection{Block,
author = {Block},
booktitle = {The Visual Story: Seeing the Structure of Film, TV, and New Media},
chapter = {3},
pages = {13--74},
title = {{Space}}
}
@incollection{Card1999,
author = {Card, S. K. and Mackinlay, J. D. and Shneiderman, B.},
booktitle = {Readings in Information Visualization: Using Vision to Think},
chapter = {1},
pages = {1--34},
publisher = {Morgan Kaufmann},
title = {{Information visualization}},
year = {1999}
}
@inproceedings{Chang2002,
author = {Chang, D. and Dooley, L. and Tuovinen, J. E.},
title = {{Gestalt Theory in Visual Screen Design ew Look at an Old Subject}},
booktitle = {Conf. Computers in Education},
pages = {1--8},
year = {2002}
}
@incollection{Cleveland1984b,
author = {Cleveland, W. S. and McGill, R.},
title = {{Introduction}},
booktitle = {The Elements of Graphing Data},
chapter = {1},
pages = {1--18},
year = {1984}
}
@incollection{Cleveland1984,
author = {Cleveland, W. S. and McGill, R.},
title = {{Graphical Perception}},
booktitle = {The Elements of Graphing Data},
chapter = {4},
doi = {10.2307/2288400},
number = {387},
pages = {221--270},
url = {http://www.jstor.org/stable/2288400?origin=crossref},
volume = {79},
year = {1984}
}
@incollection{Cutting,
author = {Cutting, J. E.},
title = {{Perceiving Scenes in Film and in the World}},
booktitle = {Moving Image Theory: Ecological Considerations},
editor = {Anderson, J. D. and Anderson, B. F.},
pages = {1--22},
url = {http://sites.google.com/site/seeadogseminar/cutting\_2005.pdf}
}
@book{Cuttinga,
author = {Cutting, J. E. and Vishton, P. M.},
title = {{Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth}},
booktitle = {Handbook of Perception and Cognition, Vol 5: Perception of space and motion},
pages = {169--117},
volume = {5}
}
@article{Feldman2003,
author = {Feldman, J.},
doi = {10.1016/S1364-6613(03)00111-6},
journal = {Trends in Cognitive Sciences},
number = {6},
pages = {252--256},
title = {{What is a visual object?}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1364661303001116},
volume = {7},
year = {2003}
}
@inproceedings{Findlater2009a,
author = {Findlater, L. and Moffatt, K. A. and McGrenere, J. and Dawson, J.},
title = {{Ephemeral adaptation: The use of gradual onset to improve menu selection performance}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1655--1664},
url = {http://portal.acm.org/citation.cfm?id=1518701.1518956},
year = {2009}
}
@article{Furnas1997,
author = {Furnas, G. W.},
title = {{Effective view navigation}},
doi = {10.1145/258549.258800},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {367--374},
url = {http://portal.acm.org/citation.cfm?doid=258549.258800},
year = {1997}
}
@incollection{Golledge,
author = {Golledge and Stimson},
title = {{Spatial Cognition, Cognitive Mapping, and Cognitive Maps}}
booktitle = {Spatial Cognition and Cognitive Mapping},
chapter = {7},
}
@article{Goodale,
author = {Goodale, M. A. and Eagleson, R.},
title = {{Constraints and principles for the design of human-machine interfaces: a virtual reality approach}},
doi = {10.1109/ICSMC.1999.823380},
journal = {Proc. IEEE Intl. Conf. Systems, Man, and Cybernetics (SMC)},
number = {6313},
pages = {1089--1094},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=823380}
}
@article{Goodale1998,
author = {Goodale, M. A. and Haffenden, A.},
title = {{Frames of reference for perception and action in the human visual system.}},
journal = {Neuroscience and biobehavioral reviews},
number = {2},
pages = {161--72},
url = {http://www.ncbi.nlm.nih.gov/pubmed/9579308},
volume = {22},
year = {1998}
}
@article{Interrante2000,
author = {Interrante, V.},
doi = {10.1109/MCG.2000.888001},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {6},
pages = {6--11},
title = {{Harnessing natural textures for multivariate visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=888001},
volume = {20},
year = {2000}
}
@article{Jelinski1996,
author = {Jelinski, D. E. and Wu, J.},
doi = {10.1007/BF02447512},
journal = {Landscape Ecology},
number = {3},
pages = {129--140},
title = {{The modifiable areal unit problem and implications for landscape ecology}},
url = {http://www.springerlink.com/index/10.1007/BF02447512},
volume = {11},
year = {1996}
}
@inproceedings{Kamba1996,
author = {Kamba, T. and Elson, S. A. and Harpold, T. and Stamper, T. and Sukaviriya, P.},
title = {{Using small screen space more efficiently}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {390},
url = {http://portal.acm.org/citation.cfm?id=238582\&amp;coll=portal\&amp;dl=ACM},
year = {1996}
}
@article{Kim2004,
author = {Kim, S. and Hagh-Shenas, H. and Interrante, V.},
doi = {10.1109/TVCG.2004.5},
title = {{Conveying shape with texture: experimental investigations of texture's effects on shape categorization judgments}},
journal = {IEEE. Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {471--83},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18579974},
volume = {10},
year = {2004}
}
@article{Kuipers1982,
author = {Kuipers, B.},
doi = {10.1177/0013916584142005},
journal = {Environment and Behavior},
number = {2},
pages = {202--220},
title = {{The Map in the Head Metaphor}},
url = {http://eab.sagepub.com/cgi/doi/10.1177/0013916584142005},
volume = {14},
year = {1982}
}
@inproceedings{Kurtenbach1993,
author = {Kurtenbach, G. and Buxton, W.},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {482--487},
title = {{The limits of expert performance using hierarchic marking menus}},
url = {http://portal.acm.org/citation.cfm?id=169426\&coll=GUIDE\&dl=GUIDE\&ret=1},
year = {1993}
}
@article{Liu2003a,
author = {Liu, G. and Healey, C. G. and Enns, J. T.},
journal = {Perception and Psychophysics},
number = {5},
pages = {678--94},
title = {{Target detection and localization in visual search: a dual systems perspective}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/12956577},
volume = {65},
year = {2003}
}
}}}
{{{
@article{Kumar,
author = {Kumar, N. and {Lolla V.}, N. and Keogh, E. and Lonardi, S. and Ratanamahatana, C. A. and Wei, L.},
title = {{Time-series bitmaps: a practical visualization tool for working with large time series databases}}
pages = {531--535},
}
@article{Ruchikachorn2015,
author = {Ruchikachorn, P. and Mueller, K.},
journal = {IEEE Trans. on Visualization and Computer Graphics (TVCG)},
title = {{Learning visualizations by analogy: Promoting visual literacy through visualization morphing}},
volume = {pre-print},
year = {2015}
}
@inproceedings{Viegas2004a,
author = {Vi\'{e}gas, F. B. and Boyd, D. and Nguyen, D. H. and Potter, J. and Donath, J.},
title = {{Digital artifacts for remembering and storytelling: posthistory and social network fragments}},
booktitle = {Hawaii Intl. Conf. System Sciences (HICSS)},
year = {2004}
}
@inproceedings{Hao2007,
author = {Hao, M. and Dayal, U. and Keim, D. A. and Schreck, T.},
title = {{Multi-resolution techniques for visual exploration of large time-series data}},
booktitle = {Proc. Eurographics / IEEE-VGTC Symp. Visualization (EuroVIs)},
pages = {27--34},
year = {2007}
}
@article{Byron2008,
author = {Byron, L. and Wattenberg, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1245--1252},
title = {{Stacked graphs - geometry \& aesthetics}},
volume = {14},
year = {2008}
}
@inproceedings{Viegas2004b,
author = {Vi\'{e}gas, F. B. and Wattenberg, M. and Dave, K.},
title = {{Studying cooperation and conflict between authors with history flow visualizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {575--582},
year = {2004}
}
@inproceedings{Matkovic2008,
author = {Matkovi\'{c}, K. and Freiler, W. and Gra\v{c}anin, D. and Hauser, H.},
title = {{ComVis: A coordinated multiple views system for prototyping new visualization technology}},
booktitle = {Proc. Intl. Conf. Information Visualisation},
pages = {215--220},
year = {2008}
}
@inproceedings{Hao2007,
author = {Hao, M. and Dayal, U. and Keim, D. A. and Schreck, T.},
title = {{Multi-resolution techniques for visual exploration of large time-series data}},
booktitle = {Proc. Eurographics / IEEE-VGTC Symp. Visualization (EuroVIs)},
pages = {27--34},
year = {2007}
}
@inproceedings{Viegas2004a,
author = {Viegas, F. B. and Boyd, D. and Nguyen, D. H. and Potter, J. and Donath, J.},
title = {{Digital artifacts for remembering and storytelling: posthistory and social network fragments}},
booktitle = {Hawaii Intl. Conf. System Sciences (HICSS)},
year = {2004}
}
@inproceedings{Vuillemot2015,
author = {Vuillemot, R. and Perin, C.},
title = {{Investigating the direct manipulation of ranking tables for time navigation}},
booktitle = {To Appear in Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {4--7},
year = {2015}
}
@article{Xie2013,
author = {Xie, Z. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Exploring large scale time-series data using nested timelines}},
journal = {SPIE-IS\&T},
pages = {865405--1--13},
volume = {8654},
year = {2013}
}
@article{Gousie2013,
author = {Gousie, Michael B. and Grady, John and Branagan, Melissa},
title = {{Visualizing trends and clusters in ranked time-series data}},
journal = {SPIE-IS\&T},
pages = {90170F--1--12},
volume = {9017},
year = {2013}
}
@article{Hao2012b,
author = {Hao, M. C. and Marwah, M. and Janetzko, H. and Dayal, U. and Keim, D. A. and Patnaik, D. and Ramakrishnan, N. and Sharma, R. K.},
title = {{Visual exploration of frequent patterns in multivariate time series}},
journal = {Information Visualization},
pages = {71--83},
volume = {11},
year = {2012}
}
@article{Elmqvist2010a,
author = {Elmqvist, N. and Fekete, J.-D.},
title = {{Hierarchical aggregation for information visualization: Overview, techniques, and design guidelines}},
journal = {IEEE Trans. on Visualization and Computer Graphics (TVCG)},
pages = {439--454},
volume = {16},
year = {2010}
}
@article{Hur2013,
author = {Hur, I and Kim, S.-H. and Samak, A. and Yi, J. S.},
title = {{A comparative study of three sorting techniques in performing cognitive tasks on a tabular representation}},
journal = {Intl. J. Human-Computer Interaction (IJHCI)},
number = {6},
pages = {379--390},
volume = {29},
year = {2013}
}
@article{Meyer2014,
author = {Meyer, M. and Sedlmair, M. and Quinan, P. S. and Munzner, T.},
title = {{The nested blocks and guidelines model}},
journal = {Information Visualization},
note = {doi.org/10.1177/1473871613510429},
volume = {In press}
}
@article{Sedlmair2012b,
author = {Sedlmair, M. and Meyer, M. and Munzner, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2431--2440},
title = {{Design study methodology: Reflections from the trenches and the stacks}},
volume = {18},
year = {2012}
}
@article{Sedlmair2011,
author = {Sedlmair, M. and Isenberg, P. and Baur, D. and Butz, A.},
journal = {Information Visualization},
number = {3},
pages = {248--266},
title = {{Information visualization evaluation in large companies: Challenges, experiences and recommendations}},
volume = {10},
year = {2011}
}
@book{Munzner2014,
author = {Munzner, T.},
pages = {1--306},
publisher = {AK Peters},
title = {{Visualization Analysis and Design}},
year = {2014}
}
@article{McKenna2014,
author = {McKenna, S. and Mazur, D. and Agutter, J. and Meyer, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2191--2200},
title = {{Design activity framework for visualization design}},
volume = {20},
year = {2014}
}
@article{Wood2014,
author = {Wood, J. and Beecham, R. and Dykes, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2171--2180},
title = {{Moving beyond sequential design: Reflections on a rich multi-channel approach to data visualization}},
volume = {20},
year = {2014}
}
@article{Brehmer2013,
author = {Brehmer, M. and Munzner, T.},
number = {12},
pages = {2376--2385},
title = {{A multi-level typology of abstract visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Correll2014b,
author = {Correll, M. and Gleicher, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2142--2151},
title = {{Error bars considered harmful: Exploring alternate encodings for mean and error}},
volume = {20},
year = {2014}
}
@article{Spitzer2014,
author = {Spitzer, M. and Wildenhain, J. and Rappsilber, J. and Tyers, M.},
title = {{BoxPlotR: A web tool for generation of box plots}},
journal = {Nature Methods},
number = {2},
pages = {121--122},
volume = {11},
year = {2014}
}
@techreport{Few2014,
author = {Few, S.},
title = {{Distribution displays, conventional and potential}},
year = {2014}
}
@techreport{Wickham2011,
title = {40 years of boxplots},
author = {Wickham, H. and Stryjewski, L.},
note = {http://vita.had.co.nz/papers/boxplots.pdf},
year = {2011}
}
@article{Booshehrian2012,
author = {Booshehrian, M. and M\"{o}ller, T. and Peterman, R. M. and Munzner, T.},
doi = {10.1111/j.1467-8659.2012.03116.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {1235--1244},
title = {{Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity In fisheries management decision making}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03116.x},
volume = {31},
year = {2012}
}
@article{Aigner2011b,
author = {Aigner, W. and Kainz, C. and Ma, R. and Miksch, S.},
title = {{Bertin was right: An empirical evaluation of indexing to compare multivariate time-series data using line plots}},
journal = {Computer Graphics Forum},
number = {1},
pages = {215--228},
volume = {30},
year = {2011}
}
@article{Isenberg2011a,
author = {Isenberg, P. and Bezerianos, A. and Dragicevic, P. and Fekete, J. D.},
journal = {IEEE TVCG (Proc. InfoVis)},
number = {12},
pages = {2469--2478},
title = {{A study on dual-scale data charts}},
volume = {17},
year = {2011}
}
@phdthesis{Monroe2014,
author = {Monroe, M.},
title = {{Interactive Event Sequence Query and Transformation}},
pages = {1--307},
school = {University of Maryland, College Park},
year = {2014}
}
@article{Bach2014,
author = {Bach, B. and Dragicevic, P. and Archambault, D. and Hurter, C. and Carpendale, S.},
title = {{A review of temporal data visualizations based on space-time cube operations}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
volume = {33},
year = {2014}
}
@inproceedings{Zhao2011a,
author = {Zhao, J. and Chevalier, F. and Balakrishnan, R.},
title = {{KronoMiner: using multi-foci navigation for the visual exploration of time-series data}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1737--1746},
year = {2011}
}
@article{Zhao2011,
author = {Zhao, J. and Chevalier, F. and Pietriga, E. and Balakrishnan, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2422--2431},
title = {{Exploratory analysis of time-series with ChronoLenses}},
volume = {17},
year = {2011}
}
@inproceedings{Javed2010,
author = {Javed, W. and Elmqvist, N.},
title = {{Stack zooming for multi-focus interaction in time-series data visualization}},
booktitle = {Proc. IEEE Pacific Visualization Symp. (PacificVis)},
pages = {33--40},
year = {2010}
}
@article{Javed2013,
author = {Javed, W. and Elmqvist, N.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {8},
pages = {1362--1274},
title = {{Stack zooming for multifocus interaction in skewed-aspect visual spaces}},
volume = {19},
year = {2013}
}
@article{Shi2012,
author = {Shi, C. and Cui, W. and Liu, S. and Xu, P. and Chen, W. and Qu, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2669--2678},
title = {{RankExplorer: Visualization of ranking changes in large time series data}},
volume = {18},
year = {2012}
}
@inproceedings{Federico2014,
author = {Federico, P. and Hoffmann, S. and Rind, A. and Aigner, W. and Miksch, S.},
title = {{Qualizon graphs: Space-efficient time-series visualization with qualitative abstractions}},
booktitle = {Proc. ACM Advanced Visual Interfaces (AVI)},
year = {2014}
}
@inproceedings{Javed2012,
author = {Javed, W. and Elmqvist, N.},
title = {{Exploring the design space of composite visualization}},
booktitle = {Proc. IEEE Pacific Visualization Symp. (PacificVis)},
doi = {10.1109/PacificVis.2012.6183556},
pages = {1--8},
year = {2012}
}
@article{Lam2007,
author = {Lam, H. and Munzner, T. and Kincaid, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {6},
pages = {1278--1285},
title = {{Overview use in multiple visual information resolution interfaces}},
volume = {13},
year = {2007}
}
@inproceedings{Perin2014,
author = {Perin, C. and Fekete, J.-D. and Vuillemot, Romain},
title = {{\`{A} table! Improving temporal navigation in soccer ranking tables}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2014}
}
@article{Katzeff2013,
author = {Katzeff, C. and Broms, L. and J\"{o}nsson, L. I. and Westholm, U. and R\"{a}s\"{a}nen, M.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {5},
pages = {1--22},
title = {{Exploring sustainable practices in workplace settings through visualizing electricity consumption}},
volume = {20},
year = {2013}
}
@inproceedings{Ford2013,
author = {Ford, R. and Karlin, B.},
title = {{Graphical displays in eco-feedback: A cognitive approach}},
booktitle = {Proc. Human-Computer Interaction International (HCII)},
pages = {486--495},
year = {2013}
}
@inproceedings{Albers2014,
author = {Albers, D. and Correll, M. and Gleicher, M.},
title = {{Task-driven evaluation of aggregation in time series visualization}},
booktitle = {To Appear in Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2014}
}
@inproceedings{Correll2012,
author = {Correll, M. and Albers, D. and Gleicher, M. and Franconeri, S.},
title = {{Comparing averages in time series data}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@article{Weaver2009,
author = {Weaver, C.},
title = {{Conjunctive visual forms}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {929--936},
volume = {15},
year = {2009}
}
@article{Stolte2003,
author = {Stolte, C. and Tang, D. and Hanrahan, P.},
title = {{Multiscale Visualization Using Data Cubes}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {176--187},
volume = {9},
year = {2003}
}
@inproceedings{Weaver2004,
author = {Weaver, C.},
title = {{Building highly-coordinated visualizations in improvise}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2004.12},
pages = {159--166},
publisher = {Ieee},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382904},
year = {2004}
}
@article{Stolte2002,
author = {Stolte, C. and Tang, D. and Hanrahan, P.},
title = {{Polaris: a system for query, analysis, and visualization of multidimensional relational databases}},
doi = {10.1109/2945.981851},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {52--65},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=981851},
volume = {8},
year = {2002}
}
@article{VandenElzen2013,
author = {van den Elzen, S. and van Wijk, J. J.},
doi = {10.1111/cgf.12106},
title = {{Small multiples, large singles: A new approach for visual data exploration}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
month = jun,
number = {3pt2},
pages = {191--200},
url = {http://doi.wiley.com/10.1111/cgf.12106},
volume = {32},
year = {2013}
}
@article{Weaver2010a,
author = {Weaver, C.},
doi = {10.1109/TVCG.2009.94},
title = {{Cross-filtered views for multidimensional visual analysis}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {192--204},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20075481},
volume = {16},
year = {2010}
}
@article{Mackinlay2007,
author = {Mackinlay, J. D. and Hanrahan, P. and Stolte, C.},
title = {{Show me: Automatic presentation for visual analysis}},
doi = {10.1109/TVCG.2007.70594},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1137--1144},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968057},
volume = {13},
year = {2007}
}
@article{Dang2013,
author = {Dang, T. N. and Anand, A. and Wilkinson, L.},
title = {{TimeSeer: Scagnostics for high-dimensional time series}},
doi = {10.1109/TVCG.2012.128},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {470--483},
url = {http://www.ncbi.nlm.nih.gov/pubmed/23307611},
volume = {19},
year = {2013}
}
@inproceedings{Dang2012,
author = {Dang, T. N. and Wilkinson, L.},
title = {{TimeSeer: Detecting interesting distributions in multiple time series data}},
booktitle = {Proc. Intl. Symp. on Visual Information Communication and Interaction (VINCI)},
pages = {43--51},
year = {2012}
}
@inproceedings{Dang2013a,
author = {Dang, T. N. and Wilkinson, L.},
title = {{TimeExplorer: Similarity search time Series by their signatures}},
booktitle = {Proc. Intl. Symp. Visual Computing (ISVC)},
year = {2013}
}
@inproceedings{Dang2014,
author = {Dang, T. N. and Wilkinson, L.},
title = {{ScagExplorer: Exploring scatterplots by their scagnostics}},
booktitle = {To appear in Proc. IEEE Pacific Visualization Symposium (PacificVis)},
year = {2014}
}
@article{Monroe2013b,
author = {Monroe, M. and Lan, R. and Lee, H. and Plaisant, C. and Shneiderman, B.},
number = {12},
pages = {2227--2236},
title = {{Temporal event sequence simplification}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Gratzl2013,
author = {Gratzl, S. and Lex, A. and Gehlenborg, N. and Pfister, H. and Streit, M.},
number = {12},
pages = {2277--2286},
title = {{LineUp: Visual analysis of multi-attribute rankings}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Goodwin2013a,
author = {Goodwin, S. and Dykes, J. and Jones, S. and Dillingham, I. and Dove, G. and Duffy, A. and Kachkaev, A. and Slingsby, A. and Wood, J.},
number = {12},
pages = {2516--2525},
title = {{Creative user-centered visualization design for energy analysts and modelers}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@inproceedings{Goodwin2012a,
author = {Goodwin, S. and Dykes, J.},
title = {{Visualizing variations in household energy consumption}},
booktitle = {Proc. Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2012}
}
@inproceedings{Goodwin2012,
author = {Goodwin, S. and Dykes, J.},
title = {{Geovisualization of household energy consumption characteristics}},
booktitle = {Proc. Posters Conf. GIS Research UK (GISRUK)},
year = {2012}
}
@article{Hao2012a,
author = {Hao, M. C. and Marwah, M. and Janetzko, H. and Dayal, U. and Keim, D. a. and Patnaik, D. and Ramakrishnan, N. and Sharma, R. K.},
doi = {10.1177/1473871611430769},
journal = {Information Visualization},
number = {1},
pages = {71--83},
title = {{Visual exploration of frequent patterns in multivariate time series}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611430769},
volume = {11},
year = {2012}
}
@article{Qiang2012,
author = {Qiang, Y. and Delafontaine, M. and Versichele, M. and {De Maeyer}, P. and {Van de Weghe}, N.},
doi = {10.1177/1473871612436775},
journal = {Information Visualization},
number = {4},
pages = {255--272},
title = {{Interactive analysis of time intervals in a two-dimensional space}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871612436775},
volume = {11},
year = {2012}
}
@incollection{Blok2000,
author = {Blok, C.},
title = {{Monitoring change: Characteristics of dynamic geo-spatial phenomena for visual exploration}},
booktitle = {Lecture Notes in Computer Science: Spatial Cognition II},
doi = {10.1007/3-540-45460-8\_2},
editor = {Freksa, C. and Habel, C. and Brauer, W. and Wender, K. F.},
number = {1999},
pages = {16--30},
publisher = {Springer Berlin Heidelberg},
volume = {1},
year = {2000}
}
@inproceedings{Mackay2009,
author = {Mackay, W. E. and {Van Kleek}, M. G. and Tabard, A.},
title = {{Interacting with temporal data}},
booktitle = {Proc. Extended Abstracts of the ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
doi = {10.1145/1520340.1520740},
pages = {4783--4786},
url = {http://portal.acm.org/citation.cfm?doid=1520340.1520740},
year = {2009}
}
@inproceedings{Perin2012,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Progressive horizon graphs: Improving small multiples visualization of time series}},
booktitle = {Posters IEEE Conf. Information Visualization (InfoVIs)},
year = {2012}
}
@inproceedings{Buono2005,
author = {Buono, P. and Aris, A. and Plaisant, C. and Khella, A. and Shneiderman, B.},
title = {{Interactive pattern search in time series}},
booktitle = {Proc. SPIE Conf. Visualization and Data Analysis (VDA)},
doi = {10.1117/12.587537},
pages = {175--186},
url = {http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=857869},
volume = {5669},
year = {2005}
}
@inproceedings{Wattenberg2002,
author = {Wattenberg, M.},
title = {{Arc diagrams: Visualizing structure in strings}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2002.1173155},
pages = {110--116},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173155},
volume = {2002},
year = {2002}
}
@inproceedings{Shimabukuro2004,
author = {Shimabukuro, M. H. and Flores, E. F. and de Oliveira, M. C. F. and Levkowitz, H.},
title = {{Coordinated views to assist exploration of spatio-temporal data: A case study}},
booktitle = {Proc. IEEE Intl. Conf. Coordinated and Multiple Views in Exploratory Visualization (CMV)},
doi = {10.1109/CMV.2004.1319531},
pages = {107--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1319531},
year = {2004}
}
@inproceedings{Shanbhag2005,
author = {Shanbhag, P. and Rheingans, P. and DesJardins, M.},
title = {{Temporal visualization of planning polygons for efficient partitioning of geo-spatial data}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFOVIS.2005.32},
pages = {211--218},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1575786},
year = {2005}
}
@inproceedings{Lammarsch2009,
author = {Lammarsch, T. and Aigner, W. and Bertone, A. and G\"{a}rtner, J. and Mayr, E. and Miksch, S. and Smuc, M.},
title = {{Hierarchical temporal patterns and interactive aggregated views for pixel-based visualizations}},
booktitle = {Proc. IEEE Intl. Conf. Information Visualization},
doi = {10.1109/IV.2009.52},
pages = {44--50},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5190873},
year = {2009}
}
@inproceedings{Brodbeck2003,
author = {Brodbeck, D. and Girardin, L.},
title = {{Trend analysis in large timeseries of high-throughput screening data using a distortion-oriented lens with semantic zooming}},
booktitle = {Proc. Poster Compendium IEEE Symp. Information Visualization (InfoVis)},
pages = {74--75},
year = {2003}
}
@inproceedings{Keim2004a,
author = {Keim, D. A. and Schneidewind, J. and Sips, M.},
title = {{CircleView: a new approach for visualizing time-related multidimensional data sets}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/989863.989891},
pages = {179--182},
url = {http://portal.acm.org/citation.cfm?doid=989863.989891},
year = {2004}
}
@inproceedings{Dachselt2006,
author = {Dachselt, R. and Weiland, M.},
title = {{TimeZoom: a flexible detail and context timeline}},
booktitle = {Proc. Extended Abstracts of the ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
doi = {10.1145/1125451.1125590},
pages = {682--687},
url = {http://dl.acm.org/citation.cfm?doid=1125451.1125590},
year = {2006}
}
@inproceedings{McLachlan2008,
author = {McLachlan, P. and Munzner, T. and Koutsofios, E. and North, S.},
title = {{LiveRAC: interactive visual exploration of system management time-series data}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1357054.1357286},
pages = {1483--1492},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357286},
year = {2008}
}
@inproceedings{Perin2013,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Interactive horizon graphs: Improving the compact visualization of multiple time series}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466441},
pages = {3217--3226},
url = {http://dl.acm.org/citation.cfm?doid=2470654.2466441},
year = {2013}
}
@inproceedings{Fuchs2013a,
author = {Fuchs, J. and Fischer, F. and Mansmann, F. and Bertini, E. and Isenberg, P.},
title = {{Evaluation of alternative glyph designs for time series data in a small multiple setting}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466443},
pages = {3237--3246},
url = {http://dl.acm.org/citation.cfm?doid=2470654.2466443},
year = {2013}
}
@inproceedings{Zhao2012a,
author = {Zhao, J. and Drucker, S. M. and Fisher, D. and Brinkman, D.},
title = {{TimeSlice: Interactive faceted browsing of timeline data}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
isbn = {9781450312875},
pages = {433--436},
year = {2012}
}
@article{Reinders2001,
author = {Reinders, F. and Post, FriF.ts H. and Spoelder, H. J.W.},
doi = {10.1007/PL00013399},
title = {{Visualization of time-dependent data with feature tracking and event detection}},
journal = {The Visual Computer},
number = {1},
pages = {55--71},
url = {http://link.springer.com/10.1007/PL00013399},
volume = {17},
year = {2001}
}
@article{Tominski2009,
author = {Tominski, C.},
title = {{Event-based concepts for user-driven visualization}},
doi = {10.1057/ivs.2009.32},
journal = {Information Visualization},
number = {1},
pages = {65--81},
publisher = {Nature Publishing Group},
url = {http://www.palgrave-journals.com/doifinder/10.1057/ivs.2009.32},
volume = {10},
year = {2009}
}
@article{Hochheiser2004,
author = {Hochheiser, H. and Shneiderman, B.},
title = {{Dynamic query tools for time series data sets: Timebox widgets for interactive exploration}},
doi = {10.1057/palgrave.ivs.9500061},
journal = {Information Visualization},
number = {1},
pages = {1--18},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500061},
volume = {3},
year = {2004}
}
@inproceedings{Holz2009,
author = {Holz, C. and Feiner, S. K.},
title = {{Relaxed selection techniques for querying time-series graphs}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
doi = {10.1145/1622176.1622217},
pages = {213--222},
url = {http://portal.acm.org/citation.cfm?doid=1622176.1622217},
year = {2009}
}
@book{Andrienko2006,
author = {Andrienko, G. and Andrienko, N.},
pages = {715},
publisher = {Springer-Verlag},
title = {{Exploratory Analysis of Spatial and Temporal Data: A Systematic Approach}},
year = {2006}
}
@inproceedings{Monroe2013,
author = {Monroe, M. and Lan, R. and Morales del Olmo, J. and Shneiderman, B. and Plaisant, C. and Millstein, J.},
title = {The challenges of specifying intervals and absences in temporal queries: a graphical language approach},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2349--2358},
url = {http://doi.acm.org/10.1145/2470654.2481325},
doi = {10.1145/2470654.2481325}
}
@book{Aigner2011a,
author = {Aigner, W. and Miksch, S. and Schumann, H. and Tominski, C.},
title = {{Visualization of Time-Oriented Data}},
doi = {10.1007/978-0-85729-079-3},
pages = {286},
publisher = {Springer},
url = {http://www.timeviz.net/},
year = {2011}
}
@inproceedings{VanWijk1999,
author = {Wijk, J. J. van and Selow, E. R. van},
title = {{Cluster and calendar based visualization of time series data}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {pages4--9},
year = {1999}
}
@article{Rodgers2011,
author = {Rodgers, J. and Bartram, L.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {2489--2497},
title = {{Exploring Ambient and Artistic Visualization for Residential Energy Use Feedback}},
volume = {17},
year = {2011}
}
@inproceedings{Froehlich2010,
author = {Froehlich, J. and Findlater, L. and Landay, J.},
title = {{The design of eco-feedback technology}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1753326.1753629},
pages = {1999--2008},
url = {http://portal.acm.org/citation.cfm?doid=1753326.1753629},
year = {2010}
}
@inproceedings{Pierce2012,
author = {Pierce, J. and Paulos, E.},
title = {{Beyond energy monitors: Interaction, energy, and emerging energy systems}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2207771},
pages = {665--674},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2207771},
year = {2012}
}
@inproceedings{Kim2009,
author = {Kim, Y. and Schmid, T. and Charbiwala, Z. M. and Srivastava, M. B.},
title = {{ViridiScope: design and implementation of a fine grained power monitoring system for homes}},
booktitle = {Proc. ACM Intl. Conf. Ubiquitous Computing (UbiComp)},
doi = {10.1145/1620545.1620582},
pages = {245--254},
url = {http://portal.acm.org/citation.cfm?doid=1620545.1620582},
year = {2009}
}
@inproceedings{Holmes2007,
author = {Holmes, T. G.},
title = {{Eco-visualization: combining art and technology to reduce energy consumption}},
booktitle = {Proc. ACM SIGCHI Conf. Creativity \& Cognition},
doi = {10.1145/1254960.1254982},
pages = {153},
url = {http://portal.acm.org/citation.cfm?doid=1254960.1254982},
year = {2007}
}
@inproceedings{Pierce2008,
author = {Pierce, J. and Odom, W. and Blevis, E.},
title = {{Energy aware dwelling: a critical survey of interaction design for eco-visualizations}},
booktitle = {Proc. Australasian Conf. Computer-Human Interaction: Designing for Habitus and Habitat (OZCHI)},
doi = {10.1145/1517744.1517746},
pages = {1--8},
url = {http://portal.acm.org/citation.cfm?doid=1517744.1517746},
year = {2008}
}
@article{Ellegard2011,
author = {Ellegard, K. and Palm, J.},
doi = {10.1016/j.apenergy.2010.11.019},
journal = {Applied Energy},
number = {5},
pages = {1920--1926},
title = {{Visualizing energy consumption activities as a tool for making everyday life more sustainable}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0306261910004824},
volume = {88},
year = {2011}
}
@article{Hargreaves2010,
author = {Hargreaves, T. and Nye, M. and Burgess, J.},
doi = {10.1016/j.enpol.2010.05.068},
journal = {Energy Policy},
number = {10},
pages = {6111--6119},
title = {{Making energy visible: A qualitative field study of how householders interact with feedback from smart energy monitors}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S030142151000460X},
volume = {38},
year = {2010}
}
@inproceedings{Erickson2013,
author = {Erickson, T. and Li, M. and Kim, Y. and Deshpande, A. and Sahu, S. and Chao, T. and Sukaviriya, P. and Naphade, M.},
title = {{The Dubuque electricity portal: Evaluation of a city-scale residential electricity consumption feedback system}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466155},
url = {http://dl.acm.org/citation.cfm?doid=2470654.2466155},
pages = {1203--1212},
year = {2013}
}
@inproceedings{Erickson2012,
author = {Erickson, T. and Podlaseck, M. and Sahu, S. and Dai, J. D. and Chao, T. and Naphade, M.},
title = {{The Dubuque water portal: Evaluation of the uptake, use and impact of residential water consumption feedback}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2207772},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2207772},
pages = {675--684},
year = {2012}
}
@incollection{Gregory2012,
author = {Gregory, M. and Shneiderman, B.},
title = {{Shape identification in temporal data sets}},
booktitle = {Expanding the Frontiers of Visual Analytics and Visualization},
editor = {Dill, J. and Earnshaw, R. and Kasik, D. and Vince, J. and Wong, P. C.},
pages = {305--321},
publisher = {Springer London},
year = {2012}
}
}}}
{{{
@inproceedings{Chen2009b,
author = {Chen, Y. and Yang, J. and Ribarsky, W.},
title = {{Toward effective insight management in visual analytics systems}},
booktitle = {Proc. IEEE Pacific Visualization Symp. (PacificVis)},
pages = {49--56},
year = {2009}
}
@inproceedings{Etemadpour2015,
author = {Etemadpour, R. and Linsen, L. and Crick, C. and Forbes, A.},
title = {{A user-centric taxonomy for multidimensional data projection tasks}},
booktitle = {Proc. Intl. Conf. Information Visualization Theory and Applications (IVAPP, VISIGRAPP)},
year = {2015}
}
@inproceedings{Rind2014,
author = {Rind, A. and Aigner, W. and Wagner, M. and Miksch, S. and Lammarsch, T.},
title = {{User tasks for evaluation: Untangling the terminology throughout visualization design and development}},
booktitle = {Proc. ACM BELIV Workshop},
year = {2014}
}
@article{Sacha2014,
author = {Sacha, D. and Stoffel, A. and Stoffel, F. and Kwon, B. and Ellis, G. and Keim, D.},
journal = {To appear in IEEE TVCG (Proc. VAST)},
title = {{Knowledge generation model for visual analytics}},
year = {2014}
}
@book{Fry2008,
title={Visualizing Data},
author={Fry, B.},
publisher={O'Reilly},
year={2008}
}
@article{Brehmer2013,
author = {Brehmer, M. and Munzner, T.},
number = {12},
pages = {2376--2385},
title = {{A multi-level typology of abstract visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Sedlmair2014,
author = {Sedlmair, M. and Heinzl, C. and Bruckner, S. and Piringer, H. and M\"{o}ller, T.},
title = {{Visual parameter space analysis: A conceptual framework}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Kindlmann2014,
author = {Kindlmann, G. and Scheidegger, C.},
title = {{An algebraic process for visualization design}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Huron2014a,
author = {Huron, S. and Jansen, Y. and Carpendale, S.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Constructing visual representations: Investigating the use of tangible tokens}},
year = {2014}
}
@article{Dougherty1982,
author = {Dougherty, J. W. D. and Keller, C. M.},
title = {{Taskonomy: A practical approach to knowledge structures}},
journal = {American Ethnologist},
month = nov,
number = {4},
pages = {763--774},
volume = {9},
year = {1982}
}
@article{Norman2005,
author = {Norman, D. A.},
title = {{Human-centered design considered harmful}},
journal = {interactions},
number = {4},
pages = {14},
volume = {12},
year = {2005}
}
@inproceedings{Saket2014a,
archivePrefix = {arXiv},
title = {{Node, node-link, and node-link-group diagrams: An evaluation}},
author = {Saket, B. and Simonetto, P. and Kobourov, S. and Borner, K.},
booktitle = {To appear in IEEE Conf. Information Visualization (Infovis)},
year = {2014}
}
@article{Mirel2014,
author = {Mirel, B. and G沧, C.},
journal = {BMC Bioinformatics},
number = {1},
pages = {117--128},
title = {{Scientists' sense making when hypothesizing about disease mechanisms from expression data and their needs for visualization support}},
volume = {15},
year = {2014}
}
@article{Saket2014,
author = {Saket, B. and Simonetto, P. and Kobourov, S.},
title = {{Group-level graph visualization taxonomy}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
volume = {33},
number = {3},
year = {2014}
}
@article{Mirel2014,
author = {Mirel, B. and G沧, C.},
journal = {BMC Bioinformatics},
number = {1},
pages = {117--128},
title = {{Scientists' sense making when hypothesizing about disease mechanisms from expression data and their needs for visualization support}},
volume = {15},
year = {2014}
}
@book{Marr1982,
author = {Marr, D.},
publisher = {W. H. Freeman and Company},
title = {{Vision: A Computational Investigation into the Human Representation and Processing of Visual Information}},
year = {1982}
}
@article{Marr1979,
author = {Marr, D. and Poggio, T.},
journal = {Proc. R. Soc. London. Series B, Biological Sciences},
pages = {301--328},
title = {{A computational theory of human stereo vision}},
volume = {204},
year = {1979}
}
@article{Ahn2014,
author = {Ahn, J.-W. and Plaisant, C. and Shneiderman, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {365--376},
title = {{A task taxonomy for network evolution analysis}},
volume = {20},
year = {2014}
}
@incollection{Pretorius2014,
year={2014},
title={Tasks for multivariate network analysis},
author={Pretorius, A. J. and Purchase, H. C. and Stasko, J. T.},
booktitle={Multivariate Network Visualization},
volume={8380},
series={Lecture Notes in Computer Science},
editor={Kerren, A. and Purchase, H. C. and Ward, M. O.},
pages={77-95}
}
@article{Yang2014,
author = {Yang, H. and Li, Y. and Zhou, M. X.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {1},
pages = {1--30},
title = {{Understand users' comprehension and preferences for composing information visualizations}},
volume = {21},
year = {2014}
}
@article{Streit2012,
author = {Streit, M. and Schulz, H. and Lex, A. and Schmalstieg, D. and Schumann, H.},
doi = {10.1109/TVCG.2011.108},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {998--1010},
title = {{Model-driven design for the visual analysis of heterogeneous data}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/21690642},
volume = {18},
year = {2012}
}
@article{Roth2013,
author = {Roth, R. E.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{An empirically-derived taxonomy of interaction primitives for interactive cartography and geovisualization}},
volume = {To Appear},
year = {2013}
}
@article{Dork2011a,
author = {D묠M. and Carpendale, S. and Williamson, C.},
doi = {10.1177/1473871611425872},
journal = {Information Visualization},
number = {1},
pages = {5--21},
title = {{Visualizing explicit and implicit relations of complex information spaces}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611425872},
volume = {11},
year = {2011}
}
@book{Meister1985,
author = {Mesiter, D.},
title = {{Behavioural Analysis and Measurement Methods}},
publisher = {Wiley},
year = {1985}
}
@inproceedings{Willett2012,
author = {Willett, W. and Heer, J. and Agrawala, M.},
title = {{Strategies for Crowdsourcing Social Data Analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@article{Heer2007,
author = {Heer, J. and Robertson, G. G.},
doi = {10.1109/TVCG.2007.70539},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1240--1247},
title = {{Animated transitions in statistical data graphics}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968070},
volume = {13},
year = {2007}
}
@article{Smith2002,
author = {Smith, K. B.},
title = {{Typologies, taxonomies, and the benefits of policy classification}},
doi = {10.1111/j.1541-0072.2002.tb02153.x},
journal = {Policy Studies Journal},
number = {3},
pages = {379--395},
url = {http://doi.wiley.com/10.1111/j.1541-0072.2002.tb02153.x},
volume = {30},
year = {2002}
}
@article{Pousman2007,
author = {Pousman, Z. and Stasko, J. T. and Mateas, M.},
title = {{Casual information visualization: depictions of data in everyday life.}},
doi = {10.1109/TVCG.2007.70541},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1145--1152},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968058},
volume = {13},
year = {2007}
}
@article{Robertson1991,
author = {Robertson, P. K.},
doi = {10.1109/38.79454},
title = {{A methodology for choosing data representations}},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {56--67},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=79454},
volume = {11},
year = {1991}
}
@incollection{Knapp1995,
year={1995},
title={A task analysis approach to the visualization of geographic data},
booktitle={Cognitive Aspects of Human-Computer Interaction for Geographic Information Systems},
volume={83},
series={NATO ASI Series},
editor={Nyerges, T. L. and Mark, D. M. and Laurini, R. and Egenhofer, M. J.},
doi={10.1007/978-94-011-0103-5_25},
url={http://dx.doi.org/10.1007/978-94-011-0103-5_25},
publisher={Springer Netherlands},
author={Knapp, L.},
pages={355-371},
}
@incollection{Blok2000,
author = {Blok, C.},
title = {{Monitoring change: Characteristics of dynamic geo-spatial phenomena for visual exploration}},
booktitle = {Lecture Notes in Computer Science: Spatial Cognition II},
doi = {10.1007/3-540-45460-8\_2},
editor = {Freksa, C. and Habel, C. and Brauer, W. and Wender, K. F.},
number = {1999},
pages = {16--30},
publisher = {Springer Berlin Heidelberg},
volume = {1},
year = {2000}
}
@article{Peuquet1994,
author = {Peuquet, D. J.},
title = {{It's about time: A conceptual framework for the representation of temporal dynamics in geographic information systems}},
journal = {Annals of the Association of American Geographers},
number = {3},
pages = {441--461},
url = {http://www.jstor.org/stable/2563777},
volume = {84},
year = {1994}
}
@inproceedings{Lammarsch2012,
author = {Lammarsch, T. and Rind, A. and Aigner, W. and Miksch, S.},
title = {{Developing an extended task framework for exploratory data analysis along the structure of time}},
booktitle = {Proc. Eurographics International Workshop on Visual Analytic (EuroVA)},
pages = {1--5},
url = {http://publik.tuwien.ac.at/files/PubDat\_207621.pdf},
year = {2012}
}
@article{Aigner2007,
author = {Aigner, W. and Miksch, S. and M⬠W. and Schumann, H. and Tominski, C.},
title = {{Visualizing time-oriented data㹳tematic view}},
doi = {10.1016/j.cag.2007.01.030},
journal = {Computers and Graphics},
number = {3},
pages = {401--409},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0097849307000611},
volume = {31},
year = {2007}
}
@book{Raskin2000,
author = {Raskin, J.},
title = {{The Humane Interface: New Directions for Designing Interactive Systems}},
publisher = {Addison-Wesley Longman},
year = {2000}
}
@book{Vicente1999,
author = {Vicente, K. J.},
publisher = {CRC Press},
title = {{Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work}},
year = {1999}
}
@inproceedings{Glueck2013,
author = {Glueck, M. and Grossman, T. and Wigdor, D.},
title = {{A model of navigation for very large data views}},
booktitle = {Proc. Graphics Interface (GI)},
year = {2013}
}
@inproceedings{Dork2011,
author = {Dͮ and Carpendale, S. and Williamson, C.},
title = {{The information flaneur: A fresh look at information seeking}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1978942.1979124},
pages = {1215--1224},
url = {http://dl.acm.org/citation.cfm?doid=1978942.1979124},
year = {2011}
}
@article{Dork2012,
author = {Dͮ and Riche, N. H. and Ramos, G. and Dumais, S.},
doi = {10.1109/TVCG.2012.252},
title = {{PivotPaths: Strolling through faceted information spaces}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2709--2718},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327277},
volume = {18},
year = {2012}
}
@inproceedings{Paterno2002,
author = {Paternnd Mancini, C. and Meniconi, S.},
title = {{ConcurTaskTrees: A diagrammatic notation for specifying task models}},
booktitle = {Proc. IFIP TC13 Intl. Conf. Human-Computer Interaction (INTERACT},
pages = {362--369},
year = {1997}
}
@book{MacEachren1995,
author = {MacEachren, A. M.},
chapter = {3},
number = {13},
pages = {80--101},
publisher = {The Guilford Press},
title = {{How Maps Work: Representation, Visualization, and Design}},
volume = {81},
year = {1995}
}
@book{Aigner2011a,
author = {Aigner, W. and Miksch, S. and Schumann, H. and Tominski, C.},
title = {{Visualization of Time-Oriented Data}},
doi = {10.1007/978-0-85729-079-3},
pages = {286},
publisher = {Springer},
url = {http://www.timeviz.net/},
year = {2011}
}
@book{Andrienko2006,
author = {Andrienko, G. and Andrienko, N.},
pages = {715},
publisher = {Springer-Verlag},
title = {{Exploratory Analysis of Spatial and Temporal Data: A Systematic Approach}},
year = {2006}
}
@inproceedings{Tang2003a,
author = {Tang, C. and McEwan, G. and Greenberg, S.},
title = {{A taxonomy of tasks and visualizations for casual interaction of multimedia histories}},
booktitle = {Proc. Graphics Interface (GI)},
pages = {225--236},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.217.2167\&rep=rep1\&type=pdf},
year = {2003}
}
@book{Tufte1997,
title={Visual Explanations: Images and Quantities, Evidence and Narrative},
author={Tufte, E. R.},
year={1997},
publisher={Graphics Press}
}
@article{Stolte2002,
author = {Stolte, C. and Tang, D. and Hanrahan, P.},
doi = {10.1109/2945.981851},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {52--65},
title = {{Polaris: a system for query, analysis, and visualization of multidimensional relational databases}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=981851},
volume = {8},
year = {2002}
}
@article{Chen2009,
author = {Chen, M. and Ebert, D. and Hagen, H. and Laramee, R. S. and van Liere, R. and Ma, K. L. and Ribarsky, W. and Scheuermann, G. and Silver, D.},
doi = {10.1109/MCG.2009.6},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {1},
pages = {12--19},
title = {{Data, information, and knowledge in Visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4736452},
volume = {29},
year = {2009}
}
@article{Ribarsky2009,
author = {Ribarsky, W. and Fisher, B. and Pottenger, W. M.},
doi = {10.1057/ivs.2009.28},
journal = {Information Visualization},
number = {4},
pages = {254--262},
publisher = {Palgrave Macmillan},
title = {{Science of analytical reasoning}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2009.28},
volume = {8},
year = {2009}
}
@incollection{Purchase2008,
author = {Purchase, H. and Andrienko, N. and Jankun-Kelly, T. J. and Ward, M. O.},
title = {{Theoretical foundations of information visualization}},
booktitle = {Lecture Notes in Computer Science: Information Visualization: Human-Centered Issues and Perspectives},
doi = {10.1007/978-3-540-70956-5\_3},
editor = {Kerren, A. and Stasko, J. T. and Fekete, J. D. and North, C.},
pages = {46--64},
url = {http://www.springerlink.com/index/8821888727204053.pdf},
volume = {4950},
year = {2008}
}
@article{Liu2010,
author = {Liu, Z. and Stasko, J. T.},
doi = {10.1109/TVCG.2010.177},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {999--1008},
title = {{Mental models, visual reasoning and interaction in information visualization: a top-down perspective}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975137},
volume = {16},
year = {2010}
}
@inproceedings{Keim2006,
author = {Keim, D. A. and Mansmann, F. and Schneidewind, J. and Ziegler, H.},
title = {{Challenges in visual data analysis}},
booktitle = {Proc. Intl. Conf. Information Visualization (IV)},
doi = {10.1109/IV.2006.31},
pages = {9--16},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1648235},
year = {2006}
}
@article{Cleveland1984,
author = {Cleveland, W. S. and McGill, R.},
journal = {J. American Statistical Association},
number = {387},
pages = {531-- 554},
title = {{Graphical perception: Theory, experimentation, and application to the development of graphical methods}},
url = {http://www.jstor.org/stable/2288400},
volume = {79},
year = {1984}
}
@article{Sedig2012,
author = {Sedig, K. and Parsons, P. and Babanski, A.},
journal = {Multimedia Processing Technologies: Special Issue on Theory and Application of Visual Analytics},
number = {1},
pages = {12--28},
title = {{Towards a characterization of interactivity in visual analytics}},
volume = {3},
year = {2012}
}
@book{Stephenson1967,
title={The Play Theory of Mass Communication},
author={Stephenson, W.},
year={1967},
publisher={University of Chicago Press}
}
@incollection{Stephenson1967a,
author = {Stephenson, W.},
title = {{Two New Theories of Mass Communication}},
booktitle = {The Play Theory of Mass Communication},
chapter = {1},
pages = {1--12},
year = {1967}
}
@incollection{Stephenson1967b,
author = {Stephenson, W.},
title = {{Play Theory}},
booktitle = {The Play Theory of Mass Communication},
chapter = {4},
pages = {45--65},
year = {1967}
}
@incollection{Stephenson1967c,
author = {Stephenson, W.},
title = {{Ludenic Theory of Newsreading}},
booktitle = {The Play Theory of Mass Communication},
chapter = {11},
pages = {147--159},
year = {1967}
}
@incollection{Stephenson1967d,
author = {Stephenson, W.},
title = {{The Play Theory of Mass Communication Broadly Considered}},
booktitle = {The Play Theory of Mass Communication},
chapter = {15},
pages = {190--206},
year = {1967}
}
@article{Mackinlay1986,
author = {Mackinlay, J. D.},
doi = {10.1145/22949.22950},
journal = {ACM Trans. Graphics},
number = {2},
pages = {110--141},
title = {{Automating the design of graphical presentations of relational information}},
url = {http://portal.acm.org/citation.cfm?doid=22949.22950},
volume = {5},
year = {1986}
}
@inproceedings{Pirolli1995,
author = {Pirolli, P. and Card, S. K.},
title = {{Information foraging in information access environments}},
booktitle = {Proc. ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
doi = {10.1145/223904.223911},
pages = {51--58},
url = {http://portal.acm.org/citation.cfm?doid=223904.223911},
year = {1995}
}
@inproceedings{Tweedie1996,
address = {New York, New York, USA},
author = {Tweedie, L. and Spence, R. and Dawkes, H. and Su, H.},
title = {{Externalising abstract mathematical models}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/238386.238587},
pages = {406--412},
url = {http://portal.acm.org/citation.cfm?doid=238386.238587},
year = {1996}
}
@article{Shneiderman2002,
author = {Shneiderman, B.},
doi = {10.1057/palgrave/ivs/9500006},
journal = {Information Visualization},
number = {1},
pages = {5--12},
title = {{Inventing discovery tools: combining information visualization with data mining}},
url = {http://www.palgrave-journals.com/doifinder/10.1057/palgrave/ivs/9500006},
volume = {1},
year = {2002}
}
@inproceedings{Kang2011,
author = {Kang, Y. A. and Stasko, J. T.},
title = {{Characterizing the Intelligence Analysis Process: Informing Visual Analytics Design through a Longitudinal Field Study}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {19--28},
volume = {17},
year = {2011}
}
@article{Kandel2011,
author = {Kandel, S. and Heer, J. and Plaisant, C. and Kennedy, J. and van Ham, F. and Riche, N. H. and Weaver, C. and Lee, B. and Brodbeck, D. and Buono, P.},
doi = {10.1177/1473871611415994},
journal = {Information Visualization},
number = {4},
pages = {271--288},
title = {{Research directions in data wrangling: Visualizations and transformations for usable and credible data}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415994},
volume = {10},
year = {2011}
}
@inproceedings{Kandel2012a,
author = {Kandel, S. and Parikh, R. and Paepcke, A. and Hellerstein, J. M. and Heer, J.},
title = {{Profiler: Integrated Statistical Analysis and Visualization for Data Quality Assessment}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
year = {2012}
}
@inproceedings{Gleicher2012,
author = {Gleicher, M.},
title = {{Stop the evaluation arms race! A call to evaluate visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Buxton1986,
author = {Buxton, W.},
title = {{Chunking and phrasing and the design of human-computer dialogues}},
booktitle = {Proc. IFIP World Computer Congress},
pages = {475--480},
year = {1986}
}
@article{Heer2008b,
author = {Heer, J. and Mackinlay, J. D. and Stolte, C. and Agrawala, M.},
doi = {10.1109/TVCG.2008.137},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1189--1196},
title = {{Graphical histories for visualization: supporting analysis, communication, and evaluation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988963},
volume = {14},
year = {2008}
}
@article{Munzner2003,
author = {Munzner, T. and GuimbretiƮ and Tasiran, S. and Zhang, L. and Zhou, Y.},
journal = {ACM Trans. Graphics (TOG)},
number = {3},
pages = {462},
title = {{TreeJuxtaposer: scalable tree comparison using Focus+ Context with guaranteed visibility}},
url = {http://portal.acm.org/citation.cfm?id=882262.882291},
volume = {22},
year = {2003}
}
@inproceedings{Grosjean2002,
author = {Grosjean, J. and Plaisant, C. and Bederson, B. B.},
title = {{SpaceTree: supporting exploration in large node link tree, design evolution and empirical evaluation}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2002.1173148},
pages = {57--64},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173148},
year = {2002}
}
@inproceedings{Kosara2003,
author = {Kosara, R. and Hauser, H. and Gresh, D.},
title = {{An interaction view on information visualization}},
booktitle = {Proc. Eurographics},
pages = {123--137},
year = {2003}
}
@article{Lee2012a,
author = {Lee, B. and Isenberg, P. and Riche, N. H. and Carpendale, S.},
doi = {10.1109/TVCG.2012.204},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2689--2698},
title = {{Beyond mouse and keyboard: Expanding design considerations for information visualization interactions}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327275},
volume = {18},
year = {2012}
}
@article{Isenberg2009,
author = {Isenberg, P. and Fisher, D.},
doi = {10.1111/j.1467-8659.2009.01444.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {1031--1038},
title = {{Collaborative brushing and linking for co-located visual analytics of document collections}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2009.01444.x},
volume = {28},
year = {2009}
}
@article{Scholtz2009a,
author = {Scholtz, J. and Cook, K. A. and Whiting, M. A. and Lemon, D. and Greenblatt, H.},
doi = {10.1057/ivs.2009.20},
journal = {Information Visualization},
number = {4},
pages = {294--301},
title = {{Visual analytics technology transition progress}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2009.20},
volume = {8},
year = {2009}
}
@article{Silva2007,
author = {Silva, C. T. and Freire, J. and Callahan, S. P.},
doi = {10.1109/MCSE.2007.106},
journal = {Computing in Science \& Engineering},
number = {5},
pages = {82--89},
title = {{Provenance for visualizations: Reproducibility and beyond}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4293741},
volume = {9},
year = {2007}
}
@article{Hoffman2004,
author = {Hoffman, R. R. and Hayes, P. J.},
doi = {10.1109/MIS.2004.1265891},
journal = {IEEE Intelligent Systems},
number = {1},
pages = {86--89},
title = {{The pleasure principle}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1265891},
volume = {19},
year = {2004}
}
@incollection{MacEachren1994,
author = {MacEachren, A. M.},
title = {{Visualization in modern cartography: setting the agenda}}.
booktitle = {Visualization in Modern Cartography},
editor = {{MacEachren, A. M. and Taylor, D. R. F.},
pages = {1--12},
publisher = {Pergamon},
year = {1994}
}
@techreport{Roth2011,
author = {Roth, R. E.},
title = {{Interacting with maps: The science and practice of carographic interaction}},
school = {Pennsylvania State University},
year = {2011}
}
@inproceedings{Aigner2011,
author = {Aigner, W},
title = {{Understanding the role and value of interaction: First steps}},
booktitle = {Proc. Intl. WS. Visual Analytics},
pages = {17--20},
year = {2011}
}
@inproceedings{Roth2012b,
author = {Roth, R. E.},
title = {{An empirically-derived taxonomy of cartographic interaction primitives}},
booktitle = {Proc. Intl. Conf. Geographic Information Science},
pages = {1--6},
year = {2012}
}
@book{Munzner2014,
author = {Munzner, T.},
title = {{Visualization Analysis and Design}}
publisher = {AK Peters},
year = {2014}
}
@incollection{Munzner2009a,
author = {Munzner, T.},
title = {{Visualization}},
booktitle = {Fundamentals of Graphics},
chapter = {27},
edition = {3rd},
editor = {Shirley, P. and Marschner, S.},
pages = {675--707},
publisher = {AK Peters},
year = {2009}
}
@inproceedings{Jennings2011,
author = {Jennings, K. E. and Simonton, D. K. and Palmer, S. E.},
title = {{Understanding Exploratory Creativity in a Visual Domain}},
booktitle = {Proc. ACM Conf. Creativity and Cognition},
number = {1},
year = {2011}
}
@book{Hutchins1995,
author = {Hutchins, E.},
title = {Cognition in the Wild},
publisher = {MIT Press},
year = {1995}
}
@article{Saracevic1997,
author = {Saracevic, T.},
journal = {American Society for Information Science},
pages = {313--327},
title = {{The stratified model of information retrieval interaction: Extension and applications}},
url = {https://comminfo.rutgers.edu/~tefko/ProcASIS1997.doc},
volume = {34},
year = {1997}
}
@article{Toms2000,
author = {Toms, E. G.},
doi = {10.1006/ijhc.1999.0345},
journal = {Intl. J. Human-Computer Studies},
number = {3},
pages = {423--452},
title = {{Understanding and facilitating the browsing of electronic text}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581999903452},
volume = {52},
year = {2000}
}
@article{Menzel1964,
author = {Menzel, H.},
journal = {The Library Quarterly},
number = {1},
pages = {4--19},
title = {{The information needs of current scientific research}},
url = {http://www.jstor.org/stable/10.2307/4305417},
volume = {34},
year = {1964}
}
@inproceedings{Pohl2010,
author = {Pohl, M. and Wiltner, S. and Miksch, S.},
title = {{Exploring Information Visualization - Describing Different Interaction Patterns}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@techreport{Sedlmair2012a,
author = {Sedlmair, M. and Brehmer, M. and Ingram, S. and Munzner, T.},
institution = {Dept. Computer Science, University of British Columbia},
title = {{Dimensionality reduction in the wild: Gaps and guidance}},
year = {2012}
}
@article{Sedlmair2012c,
author = {Sedlmair, M. and Meyer, M. and Munzner, T.},
doi = {10.1109/TVCG.2012.213},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2431--2440},
title = {{Design study methodology: Reflections from the trenches and the stacks}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327248},
volume = {18},
year = {2012}
}
@article{Munzner2009,
author = {Munzner, T.},
doi = {10.1109/TVCG.2009.111},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {921--8},
title = {{A nested model for visualization design and validation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834155},
volume = {15},
year = {2009}
}
@inproceedings{Meyer2012,
author = {Meyer, M. and Sedlmair, M. and Munzner, T.},
title = {{The four-level nested model revisited: blocks and guidelines}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Quinn2011,
author = {Quinn, A. J. and Bederson, B. B.},
title = {{Human computation: A survey and taxonomy of growing field}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1403--1412},
year = {2011}
}
@inproceedings{Horvitz1999,
author = {Horvitz, E.},
title = {{Principles of mixed-initiative user interfaces}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/302979.303030},
pages = {159--166},
url = {http://portal.acm.org/citation.cfm?doid=302979.303030},
year = {1999}
}
@article{Bertini2010,
author = {Bertini, E. and Lalanne, D.},
doi = {10.1145/1809400.1809404},
journal = {ACM SIGKDD Explorations Newsletter},
number = {2},
pages = {9},
title = {{Investigating and reflecting on the integration of automatic data analysis and visualization in knowledge discovery}},
url = {http://portal.acm.org/citation.cfm?doid=1809400.1809404},
volume = {11},
year = {2010}
}
@inproceedings{Yuen2009,
author = {Yuen, M. C. and Chen, L. J. and King, I.},
title = {{A survey of human computation systems}},
booktitle = {Proc. IEEE Intl. Conf. Computational Science and Engineering},
doi = {10.1109/CSE.2009.395},
pages = {723--728},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5283450},
year = {2009}
}
@article{Terveen1995,
author = {Terveen, L. G.},
doi = {10.1016/0950-7051(95)98369-H},
journal = {Knowledge-Based Systems},
number = {2-3},
pages = {67--81},
title = {{Overview of human-computer collaboration}},
url = {http://linkinghub.elsevier.com/retrieve/pii/095070519598369H},
volume = {8},
year = {1995}
}
@inproceedings{Card1997,
author = {Card, S. K. and Mackinlay, J. D.},
title = {{The structure of the information visualization design space}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {92--99},
year = {1997}
}
@inproceedings{Maletic2002,
author = {Maletic, J. I. and Marcus, A. and Collard, M. L.},
title = {{A task oriented view of software visualization}},
booktitle = {Proc. IEEE WS. Visualizing Software for Understanding and Analysis (VISSOFT)},
doi = {10.1109/VISSOF.2002.1019792},
pages = {32--40},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1019792},
year = {2002}
}
@inproceedings{Wiss1999,
author = {Wiss, U. and Carr, D. A.},
title = {{An empirical study of task support in 3D information visualizations}},
booktitle = {Proc. IEEE Intl. Conf. Information Visualization},
doi = {10.1109/IV.1999.781587},
pages = {392--399},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=781587},
year = {1999}
}
@incollection{Bederson2003,
author = {Bederson, B. B. and Shneiderman, B.},
title = {{Theories for understanding information visualization}},
booktitle = {The Craft of Information Visualization: Readings and Reflections},
chapter = {8},
pages = {349--351},
publisher = {Morgan-Kaufmann},
year = {2003}
}
@article{Nersessian2006,
author = {Nersessian, N. J.},
doi = {10.1177/0170840606061842},
journal = {Organization Studies},
number = {1},
pages = {125--145},
title = {{The cognitive-cultural systems of the research laboratory}},
url = {http://oss.sagepub.com/cgi/doi/10.1177/0170840606061842},
volume = {27},
year = {2006}
}
@incollection{Nersessian2008,
author = {Nersessian, N. J.},
title = {{Mental modeling in conceptual change}},
booktitle = {Handbook of Conceptual Change},
editor = {Vosniadou, S.},
pages = {1--78},
publisher = {Routledge},
year = {2008}
}
@inproceedings{Zhang1993,
author = {Zhang, J.},
title = {{The Interaction between perceptual and cognitive processes in a distributed problem solving task}},
booktitle = {Proc. Fall AAAI Symp. Games: Planning and Learning},
number = {1},
pages = {1--9},
year = {1993}
}
@article{Kirsh2006,
author = {Kirsh, D.},
doi = {10.1075/pc.14.2.06kir},
journal = {Pragmatics & Cognition},
number = {2},
pages = {249--262},
title = {{Distributed cognition: A methodological note}},
url = {http://openurl.ingenta.com/content/xref?genre=article\&issn=0929-0907\&volume=14\&issue=2\&spage=249},
volume = {14},
year = {2006}
}
@inproceedings{Craft2005,
author = {Craft, B. and Cairns, P.},
title = {{Beyond guidelines: What can we learn from the Visual Information Seeking Mantra?}},
booktitle = {Proc. IEEE Intl. Conf. Information Visualization (IV)},
doi = {10.1109/IV.2005.28},
pages = {110--118},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1509067},
year = {2005}
}
@inproceedings{Viegas2004,
author = {Vi젆. B. and Boyd, D. and Nguyen, D. H. and Potter, J. and Donath, J.},
title = {{Digital artifacts for remembering and storytelling: posthistory and social network fragments}},
booktitle = {Proc. IEEE Annual Hawaii International Conference on System Sciences (HICSS)},
doi = {10.1109/HICSS.2004.1265287},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1265287},
volume = {00},
year = {2004}
}
@article{Zhang1994,
author = {Zhang, J. and Norman, D. A.},
doi = {10.1016/0364-0213(94)90021-3},
journal = {Cognitive Science},
number = {1},
pages = {87--122},
title = {{Representations in distributed cognitive tasks}},
url = {http://doi.wiley.com/10.1016/0364-0213(94)90021-3},
volume = {18},
year = {1994}
}
@article{Klein2006a,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 1: Alternative Perspectives}},
journal = {IEEE Intelligent Systems},
number = {4},
pages = {70--73},
volume = {21},
year = {2006}
}
@article{Klein2006,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 2: A Macrocognitive Model}},
journal = {IEEE Intelligent Systems},
number = {5},
pages = {88--92},
volume = {21},
year = {2006}
}
@techreport{Rasmussen1985,
author = {Rasmussen, J. and Goodstein, L. P.},
institution = {Ris䩯nal Laboratory},
title = {{Decision support in supervisory control}},
year = {1985}
}
@article{Friel2001,
author = {Friel, S. N. and Curcio, F. R. and Bright, G. W.},
journal = {Research in Mathematics Education},
number = {2},
pages = {124--158},
title = {{Making sense of graphs: Critical factors influencing comprehension and instructional implications}},
volume = {32},
year = {2001}
}
@article{Hetzler2004,
author = {Hetzler, E. and Turner, A.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {5},
pages = {22--26},
title = {{Analysis experiences using information visualization}},
volume = {24},
year = {2004}
}
@inproceedings{Ratwani2004,
author = {Ratwani, R. M. and Trafton, J. G. and Boehm-Davis, D. A.},
title = {{From specific information extraction to inferences: A hierarchical framework of graph comprehension}},
booktitle = {Proc. Human Factors and Ergonomics Soc.},
pages = {1808--1812},
url = {http://pro.sagepub.com/content/48/16/1808.short},
year = {2004}
}
@article{Keim2010a,
author = {Keim, D. and Bak, P. and Bertini, E. and Oelke, D. and Spretke, D. and Ziegler, H.},
title = {{Advanced visual analytics interfaces}},
doi = {10.1145/1842993.1842995},
journal = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {3},
url = {http://portal.acm.org/citation.cfm?doid=1842993.1842995},
year = {2010}
}
@article{Pike2009,
author = {Pike, W. A. and Stasko, J. T. and Chang, R. and O'Connell, T. A.},
doi = {10.1057/ivs.2009.22},
journal = {Information Visualization},
number = {4},
pages = {263--274},
title = {{The science of interaction}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2009.22},
volume = {8},
year = {2009}
}
@inproceedings{Green2009a,
author = {Green, T. M. and Fisher, B.},
title = {{The personal equation of complex individual cognition during visual interface interaction}},
booktitle = {Proc. IFIP WS. HCI and Visualization (HCIV/INTERACT)},
pages = {38--57},
url = {http://www.springerlink.com/index/6N8463N1K1287477.pdf},
year = {2009}
}
@article{Pohl2012b,
author = {Pohl, M. and Wiltner, S. and Miksch, S. and Aigner, W. and Rind, A.},
doi = {10.1007/s13218-012-0167-6},
journal = {K쩣he Intelligenz},
number = {2},
pages = {151--159},
title = {{Analysing interactivity in information visualisation}},
url = {http://www.springerlink.com/index/10.1007/s13218-012-0167-6},
volume = {26},
year = {2012}
}
@article{Keim2008,
author = {Keim, D. and Andrienko, G. and Fekete, J. D. and Gî and Kohlhammer, J. and MelanǮ},
journal = {Information Visualization, LNCS 4950},
pages = {154--175},
title = {{Visual analytics: Definition, process, and challenges}},
url = {http://www.springerlink.com/index/hr44n177550m262v.pdf},
year = {2008}
}
@inproceedings{Attfield2010,
author = {Attfield, S. J. and Hara, S. K. and Wong, B. L. W.},
title = {{Sensemaking in visual analytics: Processes and challenges}},
booktitle = {Proc. Intl. Symp. Visual Analytics and Technology},
year = {2010}
}
@inproceedings{Trafton2001,
author = {Trafton, J. G. and Trickett, S. B.},
title = {{A new model of graph and visualization usage}},
booktitle = {Proc. Annual. Conf. CSS},
pages = {1048--1053},
url = {http://oai.dtic.mil/oai/oai?verb=getRecord\&metadataPrefix=html\&identifier=ADA480181},
year = {2001}
}
@article{Isenberg2012,
author = {Isenberg, P. and Fisher, D. and Paul, S. A. and Morris, M. R. and Inkpen, K. and Czerwinski, M.},
doi = {10.1109/TVCG.2011.287},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {689--702},
title = {{Co-located collaborative visual analytics around a tabletop display}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22156107},
volume = {18},
year = {2012}
}
@inproceedings{Ziemkiewicz2011a,
author = {Ziemkiewicz, C. and Crouser, R. J. and Yauilla, A. R. and Su, S. L. and Ribarsky, W. and Chang, R.},
title = {{How locus of control influences compatibility with visualization style}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2011.6102445},
pages = {81--90},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6102445},
year = {2011}
}
@article{Scaife1996,
author = {Scaife, M. and Rogers, Y.},
journal = {Int. J. Human-Computer Studies},
pages = {185--213},
title = {{External cognition: how do graphical representations work?}},
url = {http://www.dourish.com/classes/ics234bs03/15-ScaifeRogers-ExternalCog.pdf},
volume = {45},
year = {1996}
}
@inproceedings{Lipford2010,
author = {Lipford, H. R. and Stukes, F. and Dou, W. and Hawkins, M. E. and Chang, R.},
title = {{Helping users recall their reasoning process}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2010.5653598},
pages = {187--194},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5653598},
year = {2010}
}
@inproceedings{Weaver2007a,
author = {Weaver, C.},
title = {{Patterns of coordination in Improvise visualizations}},
booktitle = {Visualization and Data Analysis},
doi = {10.1117/12.704570},
edition = {64950K},
pages = {64950K--64950K--12},
url = {http://spiedigitallibrary.org/proceeding.aspx?doi=10.1117/12.704570},
volume = {6495},
year = {2007}
}
@article{Spence1998,
author = {Spence, R. and Tweedie, L.},
doi = {10.1016/S0953-5438(98)00022-8},
journal = {Interacting with Computers},
number = {2},
pages = {137--146},
title = {{The Attribute Explorer: information synthesis via exploration}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0953543898000228},
volume = {11},
year = {1998}
}
@book{Spence2007,
author = {Spence, R.},
title = {Information Visualization: Design for Interaction},
edition = {2nd},
publisher = {Prentice Hall},
year = {2007}
}
@book{Wilkinson2005,
author = {Wilkinson, L.},
title = {The Grammar of Graphics},
edition = {2nd},
publisher = {Springer},
year = {2005}
}
@article{Hollan2000,
author = {Hollan, J. and Hutchins, E. and Kirsh, D.},
doi = {10.1145/353485.353487},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {2},
pages = {174--196},
title = {{Distributed cognition: toward a new foundation for human-computer interaction research}},
url = {http://portal.acm.org/citation.cfm?doid=353485.353487},
volume = {7},
year = {2000}
}
@inproceedings{Fekete2011,
author = {Fekete, J. D. and Hemery, P. L. and Baudel, T. and Wood, J.},
title = {{Obvious: A meta-toolkit to encapsulate information visualization toolkits 場oolkit to bind them all}}
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2011.6102446},
pages = {91--100},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6102446},
year = {2011}
}
@inproceedings{Koh2011,
author = {Koh, L. C. and Slingsby, A. and Dykes, J. and Kam, T. S.},
title = {{Developing and applying a user-centered model for the design and implementation of information visualization tools}},
booktitle = {Proc. Intl. Conf. Information Visualisation},
doi = {10.1109/IV.2011.32},
pages = {90--95},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6004027},
year = {2011}
}
@inproceedings{Dadzie2009,
author = {Dadzie, A. S. and Petrelli, D.},
title = {{Visual knowledge exploration and discovery from different points of view}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2009.5333438},
pages = {227--228},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5333438},
year = {2009}
}
@inproceedings{Scholtz2006,
author = {Scholtz, J.},
title = {{Beyond usability: Evaluation aspects of visual analytic environments}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2006.261416},
pages = {145--150},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4035759},
year = {2006}
}
@inproceedings{Isenberg2008a,
author = {Isenberg, P. and Tang, A. and Carpendale, S.},
title = {{An exploratory study of visual information analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1357054.1357245},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357245},
year = {2008}
}
@article{DeOliveira2003,
author = {Oliveira, M.C.F. de and Levkowitz, H.},
doi = {10.1109/TVCG.2003.1207445},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {3},
pages = {378--394},
title = {{From visual data exploration to visual data mining: A survey}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1207445},
volume = {9},
year = {2003}
}
@article{Liu2008,
author = {Liu, Z. and Nersessian, N. and Stasko, J. T.},
doi = {10.1109/TVCG.2008.121},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1173--80},
title = {{Distributed cognition as a theoretical framework for information visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988961},
volume = {14},
year = {2008}
}
@inproceedings{Mcnamara2012,
author = {McNamara, L. A. and Orlando-Gay, N.},
title = {{Reading, sorting, marking, shuffling: Mental model formation through information foraging}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@article{Pohl2012a,
author = {Pohl, M. and Smuc, M. and Mayr, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2908--2916},
title = {{The user puzzle: Explaining the interaction with visual analytics systems}},
volume = {18},
year = {2012}
}
@inproceedings{Cottam2012a,
author = {Cottam, J. A and Lumsdaine, A. and Weaver, C.},
title = {{Watch this: A taxonomy for dynamic data visualization}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {193--202},
year = {2012}
}
@article{Crouser2012,
author = {Crouser, R. J. and Chang, R.},
doi = {10.1109/TVCG.2012.195},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2859--2868},
title = {{An affordance-based framework for human computation and human-computer collaboration}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327292},
volume = {18},
year = {2012}
}
@techreport{Merkelbach1994,
author = {Merkelbach, E. J. H. M. and Schraagen, J. M. C.},
institution = {TNO Human Factors Research Institute: Information Processing},
title = {{A framework for the analysis of cognitive tasks}},
url = {http://handle.dtic.mil/100.2/ADA285345},
year = {1994}
}
@book{Kuhn1962,
author = {Kuhn, T. S.}.,
title = {The Structure of Scientific Revolutions},
publisher = {University of Chicago Press},
edition = {4th},
year = {1962}
}
@incollection{Rensink2012,
author = {Rensink, R. A.},
pages = {1--24},
title = {{On the prospects for a science of visualization}},
year = {2012}
}
@article{Bhatt2008,
author = {Bhatt, G. D.},
journal = {Knowledge Management},
number = {1},
pages = {15--26},
title = {{Organizing knowledge in the knowledge development cycle}},
volume = {4},
year = {2008}
}
@article{Bates1989,
author = {Bates, M. J.},
journal = {Online Review},
number = {13},
pages = {407--424},
title = {{The design of browsing and berrypicking techniques for the online search interface}},
year = {1989}
}
@article{Fisher1993,
author = {Fisher, P. and Dykes, J. and Wood, J.},
journal = {Cartographic Journal},
number = {2},
pages = {136--142},
title = {{Map design and visualization}},
volume = {30},
year = {1993}
}
@article{Andrienko2007,
author = {Andrienko, G. and Andrienko, N. and Jankowski, P. and Keim, D. and Kraak, M. J. and MacEachren, A. M. and Wrobel, S.},
journal = {Intl. J. Geographical Information Science},
number = {8},
pages = {839--857},
title = {{Geovisual analytics for spatial decision support: Setting the research agenda}},
volume = {21},
year = {2007}
}
@article{Tukey1980,
author = {Tukey, J. W.},
journal = {The American Statistician},
number = {1},
pages = {23--25},
title = {{We need both exploratory and confirmatory}},
volume = {34},
year = {1980}
}
@article{Thomas2006,
author = {Thomas, J. J. and Cook, K. A.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
pages = {10--13},
title = {{A visual analytics agenda}},
year = {2006}
}
@article{Chang2009,
author = {Chang, R. and Ziemkiewicz, C. and Green, T. M. and Ribarsky, W.},
title = {{Defining insight for visual analytics}},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {2},
pages = {14--7},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19462629},
volume = {29},
year = {2009}
}
@book{Norman1988,
author = {Norman, D. A.},
publisher = {Basic Books New York},
title = {The Design of Everyday Things},
url = {http://webmail.faac.unesp.br/~paula/Paula/everydaythings.pdf},
year = {1988}
}
@article{Dibiase1990,
author = {Dibiase, D.},
journal = {Bulletin of the College of Earth and Mineral Sciences},javascript:;
title = {{Visualization in the earth sciences}},
volume = {59},
issue = {2},
pages = {13--18},
year = {1990}
}
@article{Roth2012,
author = {Roth, R. E.},
journal = {Cartographic Journal},
number = {4},
pages = {376 -- 395},
title = {{Cartographic interaction primitives: Framework and synthesis}},
volume = {49},
year = {2012}
}
@book{Beveridge1958,
author = {Beveridge, W. I. B.},
edition = {Revised ed},
publisher = {W. W. Norton and Company},
title = {{The art of scientific investigation.}},
url = {http://www.cabdirect.org/abstracts/19522203234.html},
year = {1958}
}
@book{Heuer1999,
address = {Washington, DC, USA},
author = {Heuer, R.},
title = {{Psychology of Intelligence Analysis}},
publisher = {Central Intelligence Agency},
year = {1999}
}
@article{Crampton2002,
author = {Crampton, J. W.},
doi = {10.1559/152304002782053314},
journal = {Cartography and Geographic Information Science},
number = {2},
pages = {85--98},
title = {{Interactivity types in geographic visualization}},
url = {http://openurl.ingenta.com/content/xref?genre=article\&issn=1523-0406\&volume=29\&issue=2\&spage=85},
volume = {29},
year = {2002}
}
@incollection{Edsall2008,
author = {Edsall, R. and Andrienko, G. and Andrienko, N. and Buttenfield, B.},
title = {{Interactive maps for exploring spatial data}},
booktitle = {Manual of Geographic Information Systems},
chapter = {42},
editor = {Madden, M.},
pages = {837--858},
publisher = {ASPRS},
year = {2008}
}
@article{MacEachren1999,
author = {MacEachren, A. M. and Wachovitz, M. and Edsall, R. and Haug, D. and Masters, R.},
journal = {Intl. J. Geographical Information Science},
number = {4},
pages = {311--334},
title = {{Constructing knowledge from multivariate spatiotemporal data: Integrating geographical visualization with knowledge discovery in database methods}},
volume = {13},
year = {1999}
}
@article{Andrienko2003,
author = {Andrienko, N. and Andrienko, G. and Gatalsky, P.},
doi = {10.1016/S1045-926X(03)00046-6},
journal = {Visual Languages \& Computing},
number = {6},
pages = {503--541},
title = {{Exploratory spatio-temporal visualization: An analytical review}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1045926X03000466},
volume = {14},
year = {2003}
}
@book{Bailey1994,
author = {Bailey, K. D.},
publisher = {Sage Publications},
title = {{Typologies and taxonomies: An introduction to classification}},
year = {1994}
}
@inproceedings{Bautista2006,
author = {Bautista, J. and Carenini, G.},
title = {{An integrated task-based framework for the design and evaluation of visualizations to support preferential choice}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/1133265.1133308},
pages = {217--224},
url = {http://portal.acm.org/citation.cfm?doid=1133265.1133308},
year = {2006}
}
@article{Zhang1996,
author = {Zhang, J.},
journal = {Intl. J. Human-Computer Studies},
pages = {59--74},
title = {{A representational analysis of relational information displays}},
volume = {45},
year = {1996}
}
@inproceedings{Espinosa1999,
author = {Espinosa, O. J. and Hendrickson, C. and Garrett, J. H.},
title = {{Domain analysis: A technique to design a user-centered visualization framework}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {44--52},
year = {1999}
}
@article{Mullins1991,
author = {Mullins, P. M. and Treu, S.},
journal = {SIGCHI Bulletin},
pages = {52--53},
title = {{Task protocol specification (TPS): a workstation independent specification for human-computer interaction}},
volume = {23},
year = {1991}
}
@article{Carter1986,
author = {Carter, J.},
journal = {Intl. J. Man Machine Studies},
pages = {195--292},
title = {{A taxonomy of user-oriented functions}},
volume = {24},
year = {1986}
}
@article{Gawron1989,
author = {Gawron, V. and Drury, C. G. and Czaja, S. J. and Wilkins, D. M},
journal = {Intl. J. Man Machine Studies},
pages = {643--672},
title = {{A taxonomy of independent human performance}},
volume = {31},
year = {1989}
}
@article{Kirsh1994,
author = {Kirsh, D. and Maglio, P.},
doi = {10.1207/s15516709cog1804\_1},
journal = {Cognitive Science},
number = {4},
pages = {513--549},
title = {{On distinguishing epistemic from pragmatic action}},
url = {http://doi.wiley.com/10.1207/s15516709cog1804\_1},
volume = {18},
year = {1994}
}
@book{Ware2013,
address = {San Fransisco, CA, USA},
author = {Ware, C.},
edition = {3rd},
pages = {228--237},
publisher = {Morgan Kaufmann},
title = {{Information Visualization: Perception for Design}},
year = {2013}
}
@article{Scholtz2004,
author = {Scholtz, J. and Morse, E. and Hewett, T.},
journal = {Human Performance, Situation Awareness, and Automation},
title = {{In depth observational studies of professional intelligence analysts}},
year = {2004}
}
@inproceedings{Patterson2001,
author = {Patterson, E. S. and Woods, D. D. and Tinapple, D. and Roth, E. M.},
title = {{Using cognitive task analysis (CTA) to seed design concepts for intelligence analysts under data overload}},
booktitle = {Proc. Human Factors and Ergonomics Soc.},
doi = {10.1177/154193120104500437},
number = {4},
pages = {439--443},
url = {http://pro.sagepub.com/lookup/doi/10.1177/154193120104500437},
volume = {45},
year = {2001}
}
@article{Eccles2008,
author = {Eccles, R. and Kapler, T. and Harper, R. and Wright, W.},
doi = {10.1057/palgrave.ivs.9500173},
journal = {Information Visualization},
number = {1},
pages = {3--17},
title = {{Stories in GeoTime}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500173},
volume = {7},
year = {2008}
}
@inproceedings{Hampson2005,
author = {Hampson, E. and Cowley, P.},
title = {{Instrumenting the intelligence analysis process}},
booktitle = {Proc. Intl. Conf. Intelligence Analysis},
year = {2005}
}
@inproceedings{Gotz2006b,
author = {Gotz, D. and Zhou, M. X.},
title = {{A study of information gathering and result processing in intelligence analysis}},
booktitle = {Proc. ACM WS. IUI for Intelligence Analysis},
year = {2006}
}
@inproceedings{Bavoil2005,
author = {Bavoil, L. and Callahan, S. P. and Crossno, P. J. and Freire, J. and Scheidegger, C. E. and Silva, T. and Vo, H. T.},
title = {{VisTrails: Enabling interactive multiple-view visualizations}},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {135--142},
year = {2005}
}
@article{Shrinivasan2009,
author = {Shrinivasan, Y. B. and van Wijk, J. J.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {5},
pages = {34--43},
title = {{Supporting exploration awareness in information visualization}},
volume = {29},
year = {2009}
}
@inproceedings{Shrinivasan2008,
author = {Shrinivasan, Y. B. and van Wijk, J. J.},
title = {{Supporting the analytical reasoning process in information visualization}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1237--1246},
year = {2008}
}
@article{Dou2009,
author = {Dou, W. and Jeong, Do. H. and Stukes, F. and Ribarsky, W. and Lipford, H. R. and Chang, R.},
title = {{Recovering reasoning processes from user interactions}},
doi = {10.1109/MCG.2009.49},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {52--61},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4909118},
volume = {29},
year = {2009}
}
@article{Jarvenpaa1988,
author = {Jarvenpaa, S. L. and Dickson, G. W.},
journal = {Communications of the ACM},
number = {6},
pages = {764--774},
title = {{Graphics and managerial decision making: Research based guidelines}},
volume = {37},
year = {1988}
}
@inproceedings{Casner1989,
author = {Casner, S. M. and Larkin, J. H.},
title = {{Cognitive efficiency considerations for good graphic design}},
booktitle = {Proc. Conf. Cognitive Science Soc.},
year = {1989}
}
@article{Larkin1987,
author = {Larkin, J. H. and Simon, H. A.},
journal = {Cognitive Science},
pages = {65--99},
title = {{Why a diagram is (sometimes) worth ten thousand words}},
volume = {11},
year = {1987}
}
@article{Kieras1985,
author = {Kieras, D. and Polson, P. G.},
journal = {Man-Machine Studies},
pages = {365--394},
title = {{An approach to the formal analysis of user complexity}},
volume = {22},
year = {1985}
}
@article{Koedinger1990,
author = {Koedinger, K. R. and Anderson, J. R.},
doi = {10.1207/s15516709cog1404\_2},
journal = {Cognitive Science},
number = {4},
pages = {511--550},
title = {{Abstract planning and perceptual chunks: Elements of expertise in geometry}},
url = {http://doi.wiley.com/10.1207/s15516709cog1404\_2},
volume = {14},
year = {1990}
}
@article{Mayr2011,
author = {Mayr, E. and Smuc, M. and Risku, H.},
doi = {10.1177/1473871611415987},
journal = {Information Visualization},
number = {3},
pages = {232--247},
title = {{Many roads lead to Rome: Mapping users' problem-solving strategies}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415987},
volume = {10},
year = {2011}
}
@inproceedings{Treinish1999,
author = {Treinish, L. A.},
title = {{A function-based data model for visualization}},
booktitle = {Proc. IEEE Conf. Visualization Late Breaking Hot Topics},
pages = {73--76},
year = {1999}
}
@inproceedings{Kreuseler2004,
author = {Kreuseler, M. and Nocke, T. and Schumann, H.},
title = {{A history mechanism for visual data mining}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.2},
pages = {49--56},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382890},
year = {2004}
}
@inproceedings{Jankun-Kelly2002,
author = {Jankun-Kelly, T. J. and Ma, K. L. and Gertz, M.},
title = {{A model for the visualization exploration process}},
booktitle = {Proc. IEEE Conf. Visualization},
doi = {10.1109/VISUAL.2002.1183791},
pages = {323--330},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1183791},
year = {2002}
}
@article{Rheingans2002,
author = {Rheingans, P. L.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {1},
pages = {6--10},
title = {{Are we there yet? Exploring with dynamic visualization}},
volume = {22},
year = {2002}
}
@article{Upson1989,
author = {Upson, C. and Faulhaber, T. and Laidlaw, D. and Schlegel, D. and Gurwitz, R. and van Dam, A.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {4},
pages = {30--42},
title = {{The application visualization system: A computational environment for scientific visualization}},
volume = {9},
year = {1989}
}
@article{Duke2005,
author = {Duke, D. J. and Brodlie, K. W. and Duce, D. A. and Herman, I.},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {6--9},
title = {{Do you see what I mean?}},
volume = {25},
year = {2005}
}
@book{Case2008,
author = {Case, D. O.},
publisher = {Emerald Publishing Ltd.},
title = {{Looking for Information}},
year = {2008}
}
@article{Foster2003,
author = {Foster, A. and Ford, N.},
doi = {10.1108/00220410310472518},
journal = {Documentation},
number = {3},
pages = {321--340},
title = {{Serendipity and information seeking: An empirical study}},
url = {http://www.emeraldinsight.com/10.1108/00220410310472518},
volume = {59},
year = {2003}
}
@article{Wilson1999,
author = {Wilson, T. D.},
journal = {Documentation},
number = {3},
pages = {249--270},
title = {{Models of information behaviour research}},
volume = {55},
year = {1999}
}
@article{Ellis1997,
author = {Ellis, D. and Haugan, M.},
doi = {10.1108/EUM0000000007204},
journal = {Documentation},
number = {4},
pages = {384--403},
title = {{Modelling the information seeking patterns of engineers and research scientists in an industrial environment}},
url = {http://www.emeraldinsight.com/10.1108/EUM0000000007204},
volume = {53},
year = {1997}
}
@article{Wilson2000,
author = {Wilson, T. D.},
journal = {Informing Science},
number = {2},
pages = {49--56},
title = {{Human information behavior}},
volume = {3},
year = {2000}
}
@article{Li2009,
author = {Li, Y.},
doi = {10.1002/asi},
journal = {J. Am Soc. Information Science and Technology},
number = {2},
pages = {275--291},
title = {{Exploring the relationships between work task and search task in information search}},
volume = {60},
year = {2009}
}
@inproceedings{Toms1999,
author = {Toms, E. G.},
title = {{Serendipitous information retrieval}},
booktitle = {Proc. WS. Information Seeking, Searching and Querying in Digital Libraries},
number = {1968},
pages = {1--4},
year = {1999}
}
@article{Bystrom1995,
author = {Bystr뮠and J쩮, K.},
journal = {Information Processing \& Management},
number = {2},
pages = {191--213},
title = {{Task complexity affects information seeking and use}},
volume = {31},
year = {1995}
}
@inproceedings{Ahlberg1994,
author = {Ahlberg, C. and Shneiderman, B.},
title = {{Visual information seeking: Tight coupling of dynamic query filters with starfield displays}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {313--317},
year = {1994}
}
@article{Olson2003,
author = {Olson, C. and Chi, E. H.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {3},
pages = {9--10},
title = {{ScentTrails: Integrating browsing and search on the web}},
volume = {10},
year = {2003}
}
@techreport{Roth2012a,
author = {Roth, R. E.},
title = {{Interactive mapping: What we know and what we need to know}},
pages = {1--58},
school = {University of Wisconsin-Madison},
year = {2012}
}
@article{Jankun-Kelly2007,
author = {Jankun-Kelly, T. J. and Ma, K. L. and Gertz, M.},
doi = {10.1109/TVCG.2007.28},
title = {{A model and framework for visualization exploration}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {2},
pages = {357--69},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17218751},
volume = {13},
year = {2007}
}
@article{North2011,
author = {North, C. and Saraiya, P. and Duca, K.},
doi = {10.1177/1473871611415989},
journal = {Information Visualization},
number = {3},
pages = {162--181},
title = {{A comparison of benchmark task and insight evaluation methods for information visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415989},
volume = {10},
year = {2011}
}
@article{North2006,
author = {North, C.},
title = {{Toward measuring visualization insight}},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {3},
pages = {6-9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16711210},
volume = {26},
year = {2006}
}
@inproceedings{Saraiya2010,
author = {Saraiya, P. and North, C. and Duca, K.},
title = {{Comparing benchmark task and insight evaluation methods on timeseries graph visualizations}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Saraiya2006,
author = {Saraiya, P. and North, C. and Lam, V. and Duca, K.},
doi = {10.1109/TVCG.2006.85},
title = {{An insight-based longitudinal study of visual analytics}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1511--22},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17073373},
volume = {12},
year = {2006}
}
@inproceedings{Andre2009,
author = {Andre, P. and Teevan, J. and Dumais, S. T.},
title = {{Discovery Is never by chance: Designing for (un)serendipity}},
booktitle = {Proc. ACM Conf. Creativity and Cognition},
pages = {305--314},
year = {2009}
}
@article{Klahr1999,
author = {Klahr, D. and Simon, H. A.},
journal = {Psychological Bulletin},
number = {5},
pages = {524--543},
title = {{Studies of scientific creativity: Complimentary approaches and convergent findings}},
volume = {125},
year = {1999}
}
@article{Lam2008a,
author = {Lam, H.},
doi = {10.1109/TVCG.2008.109},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{A framework of interaction costs in information visualization}},
number = {6},
pages = {1149--1156},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988958},
volume = {14},
year = {2008}
}
@article{Wickham2010,
author = {Wickham, H. and Cook, D. and Hofmann, H. and Buja, A.},
title = {{Graphical inference for InfoVis}},
doi = {10.1109/TVCG.2010.161},
issn = {1077-2626},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {973--9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975134},
volume = {16},
year = {2010}
}
@book{Thomas2005,
author = {Thomas, J. J. and Cook, K. A.},
publisher = {IEEE},
title = {{Illuminating the path: The research and development agenda for visual analytics}},
url = {http://www.mendeley.com/research/illuminating-path-research-development-agenda-visual-analytics-2/},
volume = {54},
year = {2005}
}
@inproceedings{Mayr2010,
author = {Mayr, E. and Smuc, M. and Risku, H.},
title = {{Many roads lead to Rome: Mapping users' problem-solving strategies}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2010}
}
@article{Lam2011,
author = {Lam, H. and Bertini, E. and Isenberg, P. and Plaisant, C. and Carpendale, S.},
doi = {10.1109/TVCG.2011.279},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
pages = {1--18},
title = {{Empirical studies in information visualization: Seven scenarios}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22144529},
year = {2011}
}
@inproceedings{Beaudouin-Lafon2004,
author = {Beaudouin-Lafon, M.},
title = {{Designing interaction, not interfaces}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {15--22},
year = {2004}
}
@article{Heer2007a,
author = {Heer, J. and Vi젆. B. and Wattenberg, M.},
journal = {Communications of the ACM},
number = {1},
pages = {87--97},
title = {{Voyagers and voyeurs: Supporting asynchronous collaborative visualization}},
volume = {52},
year = {2007}
}
@inproceedings{Saraiya2004,
author = {Saraiya, P. and North, C. and Duca, K.},
title = {{An evaluation of microarray visualization tools for biological insight}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.5},
pages = {1-8},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382884},
year = {2004}
}
@article{Kandel2012,
author = {Kandel, S. and Paepcke, A. and Hellerstein, J. M. and Heer, J.},
doi = {10.1109/TVCG.2012.219},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {2917--2926},
title = {{Enterprise data analysis and visualization: An interview study}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327298},
volume = {18},
year = {2012}
}
@inproceedings{Springmeyer1992,
author = {Springmeyer, R. R. and Blattner, M. M. and Max, N. L.},
title = {{A characterization of the scientific data analysis process}},
booktitle = {Proc. IEEE Visualization},
doi = {10.1109/VISUAL.1992.235203},
pages = {235--242},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=235203},
year = {1992}
}
@article{Amar2005a,
author = {Amar, R. and Stasko, J. T.},
doi = {10.1109/TVCG.2005.63},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {432--442},
title = {{Knowledge precepts for design and evaluation of information visualizations.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16138553},
volume = {11},
year = {2005}
}
@inproceedings{Casner1991a,
author = {Casner, S. M.},
title = {{BOZ: task-analytic design and evaluation of visual displays}},
booktitle = {Proc. IEEE Intl. Conf. Systems, Man, and Cybernetics (SMC)},
pages = {1195--1200},
volume = {00},
year = {1991}
}
@book{Pirolli2009,
author = {Pirolli, P.},
publisher = {Oxford University Press, Inc.},
title = {{Information Foraging Theory: Adaptive Interaction with Information}},
year = {2009}
}
@article{Mullins1993,
author = {Mullins, P. M. and Treu, S.},
doi = {http://dx.doi.org/10.1016/0953-5438(93)90016-M},
journal = {Interacting with Computers},
number = {2},
pages = {139--166},
title = {{A task-based cognitive model for user-network interaction: Defining a task taxonomy to guide the interface designer}},
volume = {5},
year = {1993}
}
@inproceedings{Chi2000,
author = {Chi, E. H.},
title = {{A taxonomy of visualization techniques using the data state reference model}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
year = {2000}
}
@book{Tukey1977,
author = {Tukey, J. W.},
publisher = {Addison-Wesley},
title = {{Exploratory Data Analysis}},
year = {1977}
}
@inproceedings{Tweedie1997,
author = {Tweedie, L.},
title = {{Characterizing interactive externalizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/258549.258803},
pages = {375--382},
url = {http://portal.acm.org/citation.cfm?doid=258549.258803},
year = {1997}
}
@inproceedings{Lohse1990,
author = {Lohse, J. and Rueter, H. and Biolsi, K. and Walker, N.},
title = {{Classifying visual knowledge representations: A foundation for visualization research}},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {131--138},
year = {1990}
}
@inproceedings{Chuah1996,
author = {Chuah, M. C. and Roth, S. F.},
title = {{On the semantics of interactive visualizations}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.1996.559213},
pages = {29--36},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=559213},
year = {1996}
}
@article{Lohse1994,
author = {Lohse, G. L. and Biolsi, K. and Walker, N. and Rueter, H. H.},
doi = {10.1145/198366.198376},
journal = {Communications of the ACM},
number = {12},
pages = {36--49},
title = {{A classification of visual representations}},
url = {http://portal.acm.org/citation.cfm?doid=198366.198376},
volume = {37},
year = {1994}
}
@article{Ham2009,
author = {Ham, F. van and Perer, A.},
doi = {10.1109/TVCG.2009.108},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {953--960},
title = {{Search, show context, expand on demand: supporting large graph exploration with degree-of-interest}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834159},
volume = {15},
year = {2009}
}
@inproceedings{Tory2004,
author = {Tory, M. and M⬠T.},
title = {{Rethinking visualization: A high-level taxonomy}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.59},
pages = {151--158},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382903},
year = {2004}
}
@inproceedings{Shneiderman1996,
author = {Shneiderman, B.},
title = {{The eyes have it: a task by data type taxonomy for information visualizations}},
booktitle = {Proc. IEEE Symposium on Visual Languages},
doi = {10.1109/VL.1996.545307},
pages = {336--343},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=545307},
year = {1996}
}
@inproceedings{Amar2004,
author = {Amar, R. and Stasko, J. T.},
title = {{A knowledge task-based framework for design and evaluation of information visualizations}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {143--149},
year = {2004}
}
@inproceedings{Roth1990,
author = {Roth, S. F. and Mattis, J.},
title = {{Data characterization for intelligent graphics presentation}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/97243.97273},
pages = {193--200},
url = {http://portal.acm.org/citation.cfm?doid=97243.97273},
year = {1990}
}
@inproceedings{Gotz2008,
author = {Gotz, D. and Zhou, M. X.},
title = {{Characterizing users' visual analytic activity for insight provenance}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2008.4677365},
pages = {123--130},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4677365},
year = {2008}
}
@inproceedings{Ziemkiewicz2012a,
author = {Ziemkiewicz, C. and Gomez, S. and Laidlaw, D. H.},
title = {{Analysis within and between graphs: Observed user strategies in immunobiology visualization}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1655--1658},
year = {2012}
}
@article{Heer2012a,
author = {Heer, J. and Shneiderman, B.},
journal = {Communications of the ACM},
pages = {1--26},
title = {{Interactive dynamics for visual analysis: A taxonomy of tools that support the fluent and flexible use of visualizations}},
year = {2012}
}
@article{Kosslyn1989,
author = {Kosslyn, S. M.},
journal = {Applied Cognitive Psychology},
pages = {185--226},
title = {{Understanding charts and graphs}},
volume = {2},
year = {1989}
}
@article{Marchionini2006,
author = {Marchionini, G.},
journal = {Communications of the ACM},
number = {4},
pages = {41--46},
title = {{From finding to understanding}},
volume = {49},
year = {2006}
}
@inproceedings{Chi1998,
author = {Chi, E. H. and Riedl, J. T.},
title = {{An operator interaction framework for visualization systems}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
pages = {63--70},
year = {1998}
}
@article{Casner1991,
author = {Casner, S. M.},
journal = {ACM Trans. Graphics (TOG)},
number = {2},
pages = {111--151},
title = {{A task-analytic approach to the automated design of graphic presentations}},
volume = {10},
year = {1991}
}
@article{Kennedy1996,
author = {Kennedy, J. B. and Mitchell, K. J. and Barclay, P. J.},
journal = {ACM SIGMOD Record},
pages = {30--34},
title = {{A framework for information visualisation}},
volume = {25},
year = {1996}
}
@inproceedings{Yi2008,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, J. A},
title = {{Understanding and characterizing insights: How do people gain insights using information visualization?}},
booktitle = {Proc. of BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV '08)},
pages = {1--6},
year = {2008}
}
@inproceedings{Dix1998,
author = {Dix, A. and Ellis, G.},
title = {{Starting simple: Adding value to static visualization through simple interaction}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {124--134},
year = {1998}
}
@inproceedings{Zhou1998,
author = {Zhou, M. X. and Feiner, S. K.},
title = {{Visual task characterization for automated visual discourse synthesis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/274644.274698},
pages = {392--399},
url = {http://portal.acm.org/citation.cfm?doid=274644.274698},
year = {1998}
}
@inproceedings{Rogowitz1993,
author = {Rogowitz, B. E. and Treinish, L. A.},
title = {{An architecture for rule-based visualization}},
booktitle = {Proc. IEEE Conf. Visualization},
doi = {10.1109/VISUAL.1993.398874},
pages = {236--243},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=398874},
year = {1993}
}
@inproceedings{Wehrend1990,
author = {Wehrend, S. and Lewis, C.},
title = {{A problem-oriented classification of visualization technique}},
booktitle = {Proc. IEEE Visualization},
pages = {139--143},
year = {1990}
}
@inproceedings{Winckler2004,
author = {Winckler, M. A. and Palanque, P. and Freitas, C. M. D. S.},
title = {{Tasks and scenario-based evaluation of information visualization techniques}},
booktitle = {Proc. International Workshop on Task Models and Diagrams for User Interface Design (TAMODIA '04)},
doi = {10.1145/1045446.1045475},
url = {http://portal.acm.org/citation.cfm?doid=1045446.1045475},
year = {2004}
}
@book{Card1999,
author = {Card, S. K. and Mackinlay, J. D. and Shneiderman, B.},
title = {Readings in Information Visualization: Using Vision to Think},
pages = {1--34},
publisher = {Morgan Kaufmann},
year = {1999}
}
@inproceedings{Lee2006,
author = {Lee, B. and Plaisant, C. and Parr, C. S. and Fekete, J. D. and Henry, N.},
title = {{Task taxonomy for graph visualization categories and subject descriptors}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2006}
}
@article{Buja1996,
author = {Buja, A. and Cook, D. and Swayne, D. F.},
journal = {Computational and Graphical Statistics},
number = {1},
pages = {78--99},
title = {{Interactive high-dimensional data visualization}},
volume = {5},
year = {1996}
}
@article{Yi2007,
author = {Yi, J. S. and Kang, Y. A. and Stasko, J. T. and Jacko, J. A.},
doi = {10.1109/TVCG.2007.70515},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1224--31},
title = {{Toward a deeper understanding of the role of interaction in information visualization.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968068},
volume = {13},
year = {2007}
}
@inproceedings{Roth1991,
author = {Roth, S. F. and Mattis, J.},
title = {{Automating the presentation of information}},
booktitle = {Proc. IEEE Conf. Artificial Intelligence Applications},
doi = {10.1109/CAIA.1991.120851},
pages = {90--97},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=120851},
year = {1991}
}
@inproceedings{Amar2005,
author = {Amar, R. and Eagan, J. and Stasko, J. T.},
title = {{Low-level components of analytic activity in information visualization}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2005.1532136},
pages = {111--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532136},
year = {2005}
}
@inproceedings{Valiati2006,
author = {Valiati, E. R. A. and Pimenta, M. S. and Freitas, C. M. D. S.},
title = {{A taxonomy of tasks for guiding the evaluation of multidimensional visualizations}},
booktitle = {Proc. of BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV '06)},
pages = {1--6},
url = {http://portal.acm.org/citation.cfm?id=1168169},
year = {2006}
}
@inproceedings{Ward2004,
author = {Ward, M. and Yang, J.},
title = {{Interaction spaces in data and information visualization}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
number = {3},
pages = {137--145},
volume = {22},
year = {2004}
}
@article{Plaisant1995,
author = {Plaisant, C. and Carr, D. and Shneiderman, B.},
journal = {IEEE Software},
number = {2},
pages = {21--32},
title = {{Image-browser taxonomy and guidelines for designers}},
volume = {12},
year = {1995}
}
@article{Keim2002,
author = {Keim, D. A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{Information visualization and visual data mining}},
number = {1},
pages = {1--8},
volume = {8},
year = {2002}
}
@inproceedings{Pirolli2005,
author = {Pirolli, P. and Card, S. K.},
title = {{The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis}},
booktitle = {Proceedings of International Conference on Intelligence Analysis},
pages = {2--4},
url = {http://scholar.google.com/scholar?hl=en\&btnG=Search\&q=intitle:The+Sensemaking+Process+and+Leverage+Points+for+Analyst+Technology+as+Identified+Through+Cognitive+Task+Analysis\#0},
volume = {2005},
year = {2005}
}
@article{Ignatius1994,
author = {Ignatius, E. and Senay, H. and Favre, J.},
journal = {Visual Language and Computing},
pages = {321--338},
title = {{An intelligent system for task-specific visualization assistance}},
volume = {5},
year = {1994}
}
@article{Morse2000,
author = {Morse, E. and Lewis, M. and Olsen, K. A.},
doi = {10.1006/ijhc.2000.0412},
journal = {International Journal of Human-Computer Studies},
number = {5},
pages = {637--662},
title = {{Evaluating visualizations: using a taxonomic guide}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904129},
volume = {53},
year = {2000}
}
@article{Trafton2000,
author = {Trafton, J. G. and Kirschenbaum, S. S. and Tsui, T. L. and Miyamoto, R. T. and Ballas, J. A. and Raymond, P. D.},
doi = {10.1006/ijhc.2000.0419},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {827--850},
title = {{Turning pictures into numbers: extracting and generating information from complex visualizations}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904191},
volume = {53},
year = {2000}
}
}}}
{{{
@inproceedings{Gabora2012,
author = {Gabora, L. and Firouzi, H.},
title = {{Society functions best with an intermediate level of creativity}},
booktitle = {Proc. Cognitive Science Society},
pages = {1578--1583},
url = {http://palm.mindmodeling.org/cogsci2012/papers/0279/paper0279.pdf},
year = {2012}
}
@inproceedings{Gabora2011,
author = {Gabora, L. and Saberi, M.},
title = {{An agent-based model of the cognitive mechanisms underlying the origins of creative cultural evolution}},
booktitle = {Proc. ACM Conf. Creativity \& Cognition (C\&C)},
doi = {10.1145/2069618.2069667},
pages = {299},
url = {http://dl.acm.org/citation.cfm?doid=2069618.2069667},
year = {2011}
}
@article{Gabora2008,
author = {Gabora, L.},
doi = {10.1016/j.cogsys.2007.05.004},
journal = {Cognitive Systems Research},
number = {1-2},
pages = {104--114},
title = {{The cultural evolution of socially situated cognition}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1389041707000368},
volume = {9},
year = {2008}
}
@inproceedings{Gabora2002,
author = {Gabora, L.},
title = {{Cognitive mechanisms underlying the creative process}},
booktitle = {Proc. ACM Conf. Creativity \& Cognition (C\&C)},
doi = {10.1145/581710.581730},
pages = {126--133},
url = {http://portal.acm.org/citation.cfm?doid=581710.581730},
year = {2002}
}
@inproceedings{Gabora1990,
author = {Gabora, L. M. and Colgan, P. W.},
title = {{A model of the mechanisms underlying exploratory behaviour}},
booktitle = {Proc. Intl. Conf. Simulation of Adaptive Behavior},
pages = {475--484},
url = {http://www.vub.ac.be/CLEA/liane/papers/meb.pdf},
year = {1990}
}
@article{Dunbar2001,
author = {Dunbar, K.},
journal = {Designing for science: Implications from Everyday, Classroom, and Professional Settings},
pages = {115--140},
title = {{What Scientific Thinking Reveals About the Nature of Cognition}},
year = {2001}
}
@article{Ben-Shakhar1998,
author = {Ben-Shakhar, G. and Bar-Hillel, M. and Bilu, Y. and Shefler, G.},
journal = {Behavioural Decision Making},
number = {4},
pages = {235--249},
title = {{Seek and Ye Shall Find: Test Results Are What You Hypothesize They Are}},
volume = {11},
year = {1998}
}
@article{Dunbar1993,
author = {Dunbar, K.},
journal = {Cognitive Science},
number = {3},
pages = {397--434},
publisher = {Elsevier},
title = {{Concept discovery in a scientific domain}},
url = {http://www.sciencedirect.com/science/article/pii/S0364021305800047},
volume = {17},
year = {1993}
}
@article{Klahr1993,
author = {Klahr, D. and Fay, A. L. and Dunbar, K.},
journal = {Cognitive psychology},
pages = {111--111},
publisher = {ACADEMIC PRESS INC},
title = {{Heuristics for Scientific Experimentation: A Developmental Study}},
url = {http://utoronto.academia.edu/documents/0088/9560/KlahrFayDunbar1993.pdf},
volume = {25},
year = {1993}
}
@article{Schunn1999,
author = {Schunn, C. D. and Anderson, J. R.},
journal = {Cognitive Science},
number = {3},
pages = {337--370},
publisher = {Elsevier},
title = {{The generality/specificity of expertise in scientific reasoning}},
url = {http://www.sciencedirect.com/science/article/pii/S0364021399000063},
volume = {23},
year = {1999}
}
@article{Qin1990,
author = {Qin, Y. and Simon, H. A.},
journal = {Cognitive Science},
number = {2},
pages = {281--312},
publisher = {Elsevier},
title = {{Laboratory Replication of Scientific Discovery Processes}},
url = {http://www.sciencedirect.com/science/article/pii/036402139090005H},
volume = {14},
year = {1990}
}
@incollection{Dunbar1997,
author = {Dunbar, K.},
title = {{How scientists think: On-line creativity and conceptual change in science}},
booktitle = {Conceptual structures and processes: Emergence, discovery , and change},
editor = {Ward, T.B. and Smith, S.M. and Vaid, J.},
pages = {461--493},
publisher = {APA Press},
url = {http://www.utsc.utoronto.ca/~dunbarlab/pubpdfs/DunbarWardbook.pdf},
year = {1997}
}
@incollection{Dunbar1995,
author = {Dunbar, K.},
title = {{How Scientists Really Reason: Scientific Reasoning in Real-World Laboratories}},
booktitle = {The nature of insight},
pages = {365--395},
publisher = {MIT Press},
url = {http://cognet.mit.edu/library/books/mitpress/0262691876/cache/chpt11.pdf},
volume = {396},
year = {1995}
}
@article{Simon1981,
author = {Simon, H. A. and Langley, P. W. and Bradshaw, G. L.},
journal = {Synthese},
number = {1},
pages = {1--27},
publisher = {Springer},
title = {{Scientific Discovery as Problem Solving}},
url = {http://www.springerlink.com/index/r218q01478094097.pdf},
volume = {47},
year = {1981}
}
@article{Wiggins2006,
author = {Wiggins, G. A.},
doi = {10.1016/j.knosys.2006.04.009},
journal = {Knowledge-Based Systems},
number = {7},
pages = {449--458},
title = {{A preliminary framework for description, analysis and comparison of creative systems}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0950705106000645},
volume = {19},
year = {2006}
}
@article{Harrington1975,
author = {Harrington, D. M.},
journal = {Journal of Personality},
number = {3},
pages = {434--454},
title = {{Effects of explicit instructions to "be creative" on the psychological meaning of divergent thinking test scores.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/1185489},
volume = {43},
year = {1975}
}
@inproceedings{Jennings2009,
author = {Jennings, K. E.},
title = {{Early results with faceted classification of "alternate uses" test responses}},
booktitle = {Proc. 7th ACM Conference on Creativity \& Cognition (C\&C '09)},
doi = {10.1145/1640233.1640312},
pages = {383--384},
url = {http://portal.acm.org/citation.cfm?doid=1640233.1640312},
year = {2009}
}
@article{Klahr1988,
author = {Klahr, D. and Dunbar, K.},
journal = {Cognitive Science},
pages = {1--55},
title = {{Dual space search during scientific reasoning}},
volume = {12},
year = {1988}
}
@article{Hills2008,
author = {Hills, T. T. and Todd, P. M. and Goldstone, R. L.},
journal = {Psychological Science},
number = {8},
pages = {802--808},
title = {{Search in external and internal spaces}},
volume = {19},
year = {2008}
}
@article{Isen1987,
author = {Isen, A. M. and Daubman, K. A. and Nowicki, G. P.},
journal = {Journal of Psychology and Social Psychology},
pages = {1122--1131},
title = {{Positive Affect facilitates creative problem solving}},
volume = {52},
year = {1987}
}
@article{Carson2003,
author = {Carson, S. H. and Peterson, J. B. and Higgins, D. M.},
journal = {Journal of personality and social psychology},
number = {3},
pages = {499--506},
title = {{Decreased Latent Inhibition is associated with increased creative achievement in high-functioning individuals}},
volume = {85},
year = {2003}
}
@article{DeBruijn2008,
author = {{De Bruijn}, O. and Spence, R.},
title = {{A New Framework for Theory-Based Interaction Design Applied to Serendipitous Information Retrieval}},
doi = {10.1145/1352782.1352787},
journal = {ACM TOCHI},
number = {1},
pages = {1--38},
url = {http://portal.acm.org/citation.cfm?doid=1352782.1352787},
volume = {15},
year = {2008}
}
@article{Shneiderman2007,
author = {Shneiderman, B.},
title = {{Creativity Support Tools: Accelerating Discovery and Innovation}},
journal = {Communications of the ACM},
pages = {20--32},
volume = {50},
number = {12},
year = {2007}
}
@inproceedings{Shneiderman1999,
author = {Shneiderman, B.},
title = {{User Interfaces for Creativity Support Tools}},
booktitle = {Proc. 3rd ACM Conference on Creativity \& Cognition (C\&C '99)},
pages = {15--22},
year = {1999}
}
@inproceedings{Candy2007,
author = {Candy, L.},
title = {{Understanding and Evaluating Creativity}},
booktitle = {Proc. 6th ACM Conference on Creativity \& Cognition (C\&C '07)},
doi = {10.1080/15710880701251443},
pages = {303--304},
year = {2007}
}
@inproceedings{Prophet2009,
author = {Prophet, J.},
title = {{One Person嶥ryday Creativity is Another's Extraordinary Insight}},
booktitle = {Proc. 7th ACM Conference on Creativity \& Cognition (C\&C '09)},
pages = {3--4},
year = {2009}
}
@inproceedings{Carroll2009,
author = {Carroll, E. A. and Latulipe, C. and Fung, R. and Terry, M. and Cheriton, D. R.},
title = {{Creativity Factor Evaluation: Towards a Standardized Survey Metric for Creativity Support}},
booktitle = {Proc. 7th ACM Conference on Creativity \& Cognition (C\&C '09)},
pages = {127--136},
year = {2009}
}
@inproceedings{Candy2009,
author = {Candy, L.},
title = {{Understanding and Evaluating Creativity}},
booktitle = {Proc. 7th ACM Conference on Creativity \& Cognition (C\&C '09)},
pages = {497--498},
year = {2009}
}
@inproceedings{Hilliges2007,
author = {Hilliges, O. and Terrenghi, L. and Boring, S. and Kim, D. and Richter, H. and Butz, A.},
title = {{Designing for Collaborative Creative Problem Solving}},
booktitle = {Proc. 6th ACM Conference on Creativity \& Cognition (C\&C '07)},
pages = {137--146},
year = {2007}
}
@article{Edmonds2005,
author = {Edmonds, E. A. and Weakley, A. and Candy, L. and Fell, M. and Knott, R. and Pauletto, S.},
journal = {International Journal of ~Human-Computer Studies},
number = {4-5},
pages = {452-481},
title = {{The studio as laboratory: combining creative practice and digital technology research}},
url = {http://www.sciencedirect.com/science/article/pii/S1071581905000698},
volume = {63},
year = {2005}
}
@article{Candy1997,
author = {Candy, L. and Edmonds, E. A.},
doi = {10.1021/jm2010173},
journal = {Design Studies},
pages = {185-194},
title = {{Supporting the creative user: a criteria-based approach to interaction design}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/21987581},
volume = {18},
year = {1997}
}
@article{Hewett2005,
author = {Hewett, T.},
doi = {10.1016/j.ijhcs.2005.04.004},
journal = {International Journal of ~Human-Computer Studies},
number = {4-5},
pages = {383-409},
title = {{Informing the design of computer-based environments to support creativity}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581905000431},
volume = {63},
year = {2005}
}
@article{Maiden2004,
author = {Maiden, N. and Gizikis, A. and Robertson, S.},
title = {{Provoking Creativity: Imagine What Your Requirements Could Be Like}},
doi = {10.1109/MS.2004.1331305},
journal = {IEEE Software},
number = {05},
pages = {68--75},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1331305},
volume = {21},
year = {2004}
}
}}}
{{{
@inproceedings{Quinn2011,
author = {Quinn, A. J. and Bederson, B. B.},
title = {{Human computation: A survey and taxonomy of growing field}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1403--1412},
year = {2011}
}
@inproceedings{Horvitz1999,
author = {Horvitz, E.},
title = {{Principles of mixed-initiative user interfaces}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/302979.303030},
pages = {159--166},
url = {http://portal.acm.org/citation.cfm?doid=302979.303030},
year = {1999}
}
@inproceedings{Yuen2009,
author = {Yuen, M. C. and Chen, L. J. and King, I.},
title = {{A survey of human computation systems}},
booktitle = {Proc. IEEE Intl. Conf. Computational Science and Engineering},
doi = {10.1109/CSE.2009.395},
pages = {723--728},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5283450},
year = {2009}
}
@article{Terveen1995,
author = {Terveen, L. G.},
doi = {10.1016/0950-7051(95)98369-H},
journal = {Knowledge-Based Systems},
number = {2-3},
pages = {67--81},
title = {{Overview of human-computer collaboration}},
url = {http://linkinghub.elsevier.com/retrieve/pii/095070519598369H},
volume = {8},
year = {1995}
}
@inproceedings{Fish1992,
author = {Fish, Robert S. and Kraut, Robert E. and Root, Robert W. and Rice, Ronald E.},
title = {{Evaluating video as a technology for informal communication}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '92)},
doi = {10.1145/142750.142755},
pages = {37--48},
url = {http://portal.acm.org/citation.cfm?doid=142750.142755},
year = {1992}
}
@inproceedings{Mantei1991,
author = {Mantei, M. and Baecker, R. M. and Sellen, A. J. and Buxton, W. A. and Milligan, T. and Wellman, B.},
title = {{Experiences in the Use of a Media Space}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '91)},
pages = {203--208},
year = {1991}
}
@inproceedings{Dourish1992,
author = {Dourish, P. and Bly, S.},
title = {{Portholes: Supporting Awareness in a Distributed Group Work}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '92)},
pages = {541--547},
year = {1992}
}
@article{Olson1991,
author = {Olson, M and Bly, S.},
title = {{The Portland Experience: a report on a distributed research group}},
doi = {10.1016/0020-7373(91)90042-6},
journal = {International Journal of Man-Machine Studies},
number = {2},
pages = {211--228},
url = {http://linkinghub.elsevier.com/retrieve/pii/0020737391900426},
volume = {34},
year = {1991}
}
@inproceedings{Whittaker1994,
author = {Whittaker, S. and Frohlich, D. and Daly-Jones, O.},
title = {{Informal Workplace Communication: What is it like and how might we support it?}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '94)},
pages = {131--137},
year = {1994}
}
@inproceedings{Tang2003,
author = {Tang, C. and Mcewan, G. and Greenberg, S.},
title = {{A Taxonomy of Tasks and Visualizations for Casual Interaction of Multimedia Histories}},
booktitle = {Proc. of Graphics Interface (GI '03)},
pages = {225--236},
year = {2003}
}
}}}
{{{
@techreport{Few2010,
author = {Few, S.},
booktitle = {Perceptual Edge},
title = {{Unit charts are for kids}},
year = {2010}
}
@incollection{Tufte1983,
author = {Tufte, E. R.},
title = {{Aesthetics and technique in data graphical design}},
booktitle = {The Visual Display of Quantitative Information},
chapter = {9},
pages = {177--191},
year = {1983}
}
@inproceedings{Kazi2014,
author = {Kazi, R. H. and Chevalier, F. and Grossman, T. and Zhao, S. and Fitzmaurice, G.},
title = {{DRACO: Bringing life to illustrations with kinetic textures}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {351--360},
year = {2014}
}
@inproceedings{Grossman2015,
author = {Grossman, T. and Chevalier, F.},
title = {{Your paper is dead! Bringing life to research articles with animated figures}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (alt.chi)},
year = {2015}
}
@article{Krstajic2011,
author = {Krstaji\'{c}, M. and Bertini, E.nrico and Keim, D. A.},
title = {{Cloudlines: Compact display of event episodes in multiple time-series}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2432--2439},
volume = {17},
year = {2011}
}
@article{Slingsby2010,
author = {Slingsby, A. and Wood, J. and Dykes, J.},
title = {{Treemap cartography for showing spatial and temporal traffic patterns}},
journal = {Maps},
pages = {135--146},
year = {2010}
}
@inproceedings{Brosz2013,
author = {Brosz, J and Nacenta, M. A. and Pusch, R. and Carpendale, S. and Hurter, C.},
title = {{Transmogrification: Causal manipulation of visualizations}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
pages = {79--106},
year = {2013}
}
@inproceedings{Aseniero2015,
author = {Aseniero, B. A. and Wun, T. and Ledo, D. and Ruhe, G. and Tang, A. and Carpendale, S.},
title = {{STRATOS: Using visualization to support decisions in strategic software release planning}},
booktitle = {To Appear in Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2015}
}
@inproceedings{Vuillemot2015,
author = {Vuillemot, R. and Perin, C.},
title = {{Investigating the direct manipulation of ranking tables for time navigation}},
booktitle = {To Appear in Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {4--7},
year = {2015}
}
@article{Xie2013,
author = {Xie, Z. and Ward, M. O. and Rundensteiner, E. A.},
title = {{Exploring large scale time-series data using nested timelines}},
journal = {SPIE-IS\&T},
pages = {865405--1--13},
volume = {8654},
year = {2013}
}
@article{Gousie2013,
author = {Gousie, Michael B. and Grady, John and Branagan, Melissa},
title = {{Visualizing trends and clusters in ranked time-series data}},
journal = {SPIE-IS\&T},
pages = {90170F--1--12},
volume = {9017},
year = {2013}
}
@article{Hao2012b,
author = {Hao, M. C. and Marwah, M. and Janetzko, H. and Dayal, U. and Keim, D. A. and Patnaik, D. and Ramakrishnan, N. and Sharma, R. K.},
title = {{Visual exploration of frequent patterns in multivariate time series}},
journal = {Information Visualization},
pages = {71--83},
volume = {11},
year = {2012}
}
@article{Elmqvist2010a,
author = {Elmqvist, N. and Fekete, J.-D.},
title = {{Hierarchical aggregation for information visualization: Overview, techniques, and design guidelines}},
journal = {IEEE Trans. on Visualization and Computer Graphics (TVCG)},
pages = {439--454},
volume = {16},
year = {2010}
}
@inproceedings{Alper2014,
author = {Alper, B. and Riche, N. H. and Tobias, H.},
title = {{Structuring the space: A study on enriching node-link diagrams with visual references}},
booktitle = {Proc. ACM CHI},
pages = {1825--1834},
year = {2014}
}
@article{Heer2010b,
author = {Heer, F. and Bostock, M.},
title = {{Declarative language design for interactive visualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis))},
number = {6},
pages = {1149--1156},
volume = {16},
year = {2010}
}
@inproceedings{Roth1994,
author = {Roth, S. F. and Kolojejchick, J. and Mattis, J. and Goldstein, J.},
title = {{Interactive graphic design using automatic presentation knowledge}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {112--117},
year = {1994}
}
@inproceedings{Guimbretiere2000,
author = {Guimbret貥, F. and Winograd, T.},
title = {{FlowMenu: combining command, text, and data entry}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
pages = {213--216},
volume = {2},
year = {2000}
}
@article{Cui2014,
author = {Cui, W. and Liu, S. and Wu, Z. and Wei, H.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{How hierarchical topics evolve in large text corpora}},
year = {2014}
}
@article{Mirzargar2014,
author = {Mirzargar, M. and Whitaker, R. and Kirby, R.},
journal = {To appear in IEEE TVCG (Proc. SciVis)},
title = {{Curve boxplot: Generalization of boxplot for ensembles of curves}},
year = {2014}
}
@article{Stolper2014,
author = {Stolper, C. D. and Kahng, M. and Lin, Z. and Foerster, F. and Goel, A. and Stasko, J. and Chau, D. H.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{GLO-STIX: Graph-level operations for specifying techniques and interactive exploration}},
year = {2014}
}
@article{VandenElzen2014,
author = {van den Elzen, S. and van Wijk, J. J.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{Multivariate network exploration and presentation: From detail to overview via selections and aggregations}},
year = {2014}
}
@article{Perin2014,
author = {Perin, C. and Dragicevic, P. and Fekete, J.-D.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{Revisiting Bertin matrices: New interactions for crafting tabular visualizations}},
year = {2014}
}
@inproceedings{Alexander2014,
author = {Alexander, E. and Kohlmann, J. and Valenza, R. and Witmore, M. and Gleicher, M.},
title = {{Serendip: Topic model-driven visual exploration of text corpora}},
booktitle = {To appear in IEEE VAST},
year = {2014}
}
@article{Spitzer2014,
author = {Spitzer, M. and Wildenhain, J. and Rappsilber, J. and Tyers, M.},
title = {{BoxPlotR: A web tool for generation of box plots}},
journal = {Nature Methods},
number = {2},
pages = {121--122},
volume = {11},
year = {2014}
}
@article{Gratzl2014,
author = {Gratzl, S. and Gehlenborg, N. and Lex, A. and Pfister, H. and Streit, M.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Domino: Extracting, comparing, and manipulating subsets across multiple tabular datasets}},
year = {2014}
}
@inproceedings{Goodwin2014,
author = {Goodwin, S. and Dykes, J. and Slingsby, A.},
title = {{Visualizing the effects of scale and geography in multivariate comparison}},
booktitle = {To Appear in IEEE VAST Poster Proceedings},
year = {2014}
}
@inproceedings{Huron2014,
author = {Huron, S. and Carpendale, S. and Thudt, A. and Tang, A. and Mauerer, M.},
title = {{Constructive visualization}},
booktitle = {Proc. ACM Conf. Designing Interactive Systems (DIS)},
pages = {433--442},
volume = {2014},
year = {2014}
}
@inproceedings{Alper2013,
author = {Alper, B. and Bach, B. and {Henry Riche}, N. and Isenberg, T. and Fekete, J. D.},
title = {{Weighted graph comparison techniques for brain connectivity analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {483--492},
year = {2013}
}
@article{Jianu2014,
author = {Jianu, R. and Rusu, A. and Hu, Y. and Taggart, D.},
title = {{How to display group information on node-link diagrams: An evaluation}},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {In Press},
volume = {20},
year = {2014}
}
@article{Satyanarayan2014,
author = {Satyanarayan, A. and Heer, J.},
title = {{Authoring narrative visualizations with ellipsis}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
volume = {33},
year = {2014}
}
@inproceedings{Bigelow2014,
author = {Bigelow, A. and Drucker, S. and Fisher, D. and Meyer, M.},
title = {{Reflections on how designers design with data}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
year = {2014}
}
@book{Fry2008,
title={Visualizing Data},
author={Fry, B.},
publisher={O'Reilly},
year={2008}
}
@inproceedings{Kandel2011a,
title = {Wrangler: Interactive visual specification of data transformation scripts},
author = {Kandel, S. and Paepcke, A. and Hellerstein, J. and Heer, J.},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {3363--3372},
year = {2011}
}
@article{Moody2009,
author = {Moody, D.},
doi = {10.1109/TSE.2009.67},
journal = {IEEE Trans. Software Engineering},
number = {6},
pages = {756--779},
title = {{The physics of notations: Toward a scientific basis for constructing visual notations in software engineering}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5353439},
volume = {35},
year = {2009}
}
@article{Green1989,
author = {Green, T. R. G.},
journal = {People and Computers},
pages = {443--460},
title = {{Cognitive dimensions of notations}},
volume = {V},
year = {1989}
}
@techreport{Venolia2007,
author = {Venolia, G. and Morris, M. R. and Morris, D.},
title = {{Exploring and investigating: Supporting high-level search activities}},
institution = {Microsoft Research},
pages = {1--7},
year = {2007}
}
@incollection{Maalej2013,
author = {Maalej, W. and Thurimella, A. K.},
title = {{An Introduction to Requirements Knowledge}},
booktitle = {Managing Requirements Knowledge},
chapter = {1},
editor = {Maalej, W. and Thurimella, A. K.},
pages = {1--20},
publisher = {Springer-Verlag Berlin Heidelberg},
url = {http://link.springer.com/10.1007/978-3-642-34419-0},
year = {2013}
}
@article{McInerny2014,
author = {McInerny, G. J. and Chen, M. and Freeman, R. and Gavaghan, D. and Meyer, M. and Rowland, F. and Spiegelhalter, D. J. and Stefaner, M. and Tessarolo, G. and Hortal, J.},
doi = {10.1016/j.tree.2014.01.003},
journal = {Trends in Ecology \& Evolution},
pages = {1--10},
title = {{Information visualisation for science and policy: engaging users and avoiding bias}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0169534714000160},
year = {2014}
}
@article{Sutcliffe2014,
author = {Sutcliffe, A. and de Bruijn, O. and Thew, S. and Buchan, I. and Jarvis, P. and McNaught, J. and Procter, R.},
doi = {10.1177/1473871612445832},
journal = {Information Visualization},
number = {1},
pages = {3--17},
title = {{Developing visualization-based decision support tools for epidemiology}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871612445832},
volume = {13},
year = {2014}
}
@article{Meyer2014,
author = {Meyer, M. and Sedlmair, M. and Quinan, P. S. and Munzner, T.},
doi = {10.1177/1473871613510429},
journal = {Information Visualization},
title = {{The nested blocks and guidelines model}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871613510429},
year = {To appear}
}
@techreport{Renoust2013,
author = {Renoust, B.},
title = {{Analysis and Visualisation of Edge Entanglement in Multiplex Networks}},
school = {University of Bordeaux},
year = {2013}
}
@article{Kirby2013,
author = {Kirby, R. M. and Meyer, M.},
title = {{Visualization collaborations: What works and why}},
journal = {IEEE Trans. Computer Graphics and Applications (CG\&A)},
number = {6},
pages = {82--88},
volume = {33},
year = {2013}
}
@techreport{Gomez2013,
author = {G庬 J. A. G.},
school = {University of Maryland, College Park},
title = {{Exploring differences in multivariate datasets using hierarchies: An interactive information visualization approach}},
year = {2013}
}
@inproceedings{Grammel2013,
author = {Grammel, L. and Bennett, C. and Tory, M. and Storey, M. A.},
title = {{A survey of visualization construction user interfaces}},
booktitle = {Proc. Eurographics Conf. Visualization (EuroVis)},
year = {2013}
}
@inproceedings{Peltonen2013,
author = {Peltonen, J. and Sandholm, M. and Kaski, S.},
title = {{Information retrieval perspective to interactive data visualization}},
booktitle = {Proc. Eurographics Conf. Visualization (EuroVis)},
year = {2013}
}
@inproceedings{Engel2013,
author = {Engel, D. and Hummel, M. and Hoepel, F. and Bein, K. and Wexler, A. and Garth, C. and Harmann, B. and Hagen, H.},
title = {{Towards high-dimensional data analysis in air quality research}},
booktitle = {Proc. Eurographics Conf. Visualization (EuroVis)},
number = {3},
volume = {32},
year = {2013}
}
@inproceedings{Mackay2009,
author = {Mackay, W. E. and {Van Kleek}, M. G. and Tabard, A.},
title = {{Interacting with temporal data}},
booktitle = {Proc. Extended Abstracts of the ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
doi = {10.1145/1520340.1520740},
pages = {4783--4786},
url = {http://portal.acm.org/citation.cfm?doid=1520340.1520740},
year = {2009}
}
@inproceedings{Perin2012,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Progressive horizon graphs: Improving small multiples visualization of time series}},
booktitle = {Posters IEEE Conf. Information Visualization (InfoVIs)},
year = {2012}
}
@inproceedings{Buono2005,
author = {Buono, P. and Aris, A. and Plaisant, C. and Khella, A. and Shneiderman, B.},
title = {{Interactive pattern search in time series}},
booktitle = {Proc. SPIE Conf. Visualization and Data Analysis (VDA)},
doi = {10.1117/12.587537},
pages = {175--186},
url = {http://proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=857869},
volume = {5669},
year = {2005}
}
@inproceedings{Wattenberg2002,
author = {Wattenberg, M.},
title = {{Arc diagrams: Visualizing structure in strings}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2002.1173155},
pages = {110--116},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173155},
volume = {2002},
year = {2002}
}
@inproceedings{Shimabukuro2004,
author = {Shimabukuro, M. H. and Flores, E. F. and de Oliveira, M. C. F. and Levkowitz, H.},
title = {{Coordinated views to assist exploration of spatio-temporal data: A case study}},
booktitle = {Proc. IEEE Intl. Conf. Coordinated and Multiple Views in Exploratory Visualization (CMV)},
doi = {10.1109/CMV.2004.1319531},
pages = {107--117},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1319531},
year = {2004}
}
@inproceedings{Shanbhag2005,
author = {Shanbhag, P. and Rheingans, P. and DesJardins, M.},
title = {{Temporal visualization of planning polygons for efficient partitioning of geo-spatial data}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFOVIS.2005.32},
pages = {211--218},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1575786},
year = {2005}
}
@inproceedings{Lammarsch2009,
author = {Lammarsch, T. and Aigner, W. and Bertone, A. and G\"{a}rtner, J. and Mayr, E. and Miksch, S. and Smuc, M.},
title = {{Hierarchical temporal patterns and interactive aggregated views for pixel-based visualizations}},
booktitle = {Proc. IEEE Intl. Conf. Information Visualization},
doi = {10.1109/IV.2009.52},
pages = {44--50},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5190873},
year = {2009}
}
@inproceedings{Brodbeck2003,
author = {Brodbeck, D. and Girardin, L.},
title = {{Trend analysis in large timeseries of high-throughput screening data using a distortion-oriented lens with semantic zooming}},
booktitle = {Proc. Poster Compendium IEEE Symp. Information Visualization (InfoVis)},
pages = {74--75},
year = {2003}
}
@inproceedings{Keim2004a,
author = {Keim, D. A. and Schneidewind, J. and Sips, M.},
title = {{CircleView: a new approach for visualizing time-related multidimensional data sets}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/989863.989891},
pages = {179--182},
url = {http://portal.acm.org/citation.cfm?doid=989863.989891},
year = {2004}
}
@inproceedings{Dachselt2006,
author = {Dachselt, R. and Weiland, M.},
title = {{TimeZoom: a flexible detail and context timeline}},
booktitle = {Proc. Extended Abstracts of the ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
doi = {10.1145/1125451.1125590},
pages = {682--687},
url = {http://dl.acm.org/citation.cfm?doid=1125451.1125590},
year = {2006}
}
@inproceedings{McLachlan2008,
author = {McLachlan, P. and Munzner, T. and Koutsofios, E. and North, S.},
title = {{LiveRAC: interactive visual exploration of system management time-series data}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/1357054.1357286},
pages = {1483--1492},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357286},
year = {2008}
}
@inproceedings{Perin2013,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Interactive horizon graphs: Improving the compact visualization of multiple time series}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466441},
pages = {3217--3226},
url = {http://dl.acm.org/citation.cfm?doid=2470654.2466441},
year = {2013}
}
@inproceedings{Fuchs2013a,
author = {Fuchs, J. and Fischer, F. and Mansmann, F. and Bertini, E. and Isenberg, P.},
title = {{Evaluation of alternative glyph designs for time series data in a small multiple setting}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2470654.2466443},
pages = {3237--3246},
url = {http://dl.acm.org/citation.cfm?doid=2470654.2466443},
year = {2013}
}
@article{Reinders2001,
author = {Reinders, F. and Post, FriF.ts H. and Spoelder, H. J.W.},
doi = {10.1007/PL00013399},
title = {{Visualization of time-dependent data with feature tracking and event detection}},
journal = {The Visual Computer},
number = {1},
pages = {55--71},
url = {http://link.springer.com/10.1007/PL00013399},
volume = {17},
year = {2001}
}
@article{Tominski2009,
author = {Tominski, C.},
title = {{Event-based concepts for user-driven visualization}},
doi = {10.1057/ivs.2009.32},
journal = {Information Visualization},
number = {1},
pages = {65--81},
url = {http://www.palgrave-journals.com/doifinder/10.1057/ivs.2009.32},
volume = {10},
year = {2009}
}
@article{Hochheiser2004,
author = {Hochheiser, H. and Shneiderman, B.},
title = {{Dynamic query tools for time series data sets: Timebox widgets for interactive exploration}},
doi = {10.1057/palgrave.ivs.9500061},
journal = {Information Visualization},
number = {1},
pages = {1--18},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500061},
volume = {3},
year = {2004}
}
@inproceedings{Holz2009,
author = {Holz, C. and Feiner, S. K.},
title = {{Relaxed selection techniques for querying time-series graphs}},
booktitle = {Proc. ACM Symp. User Interface Software and Technology (UIST)},
doi = {10.1145/1622176.1622217},
pages = {213--222},
url = {http://portal.acm.org/citation.cfm?doid=1622176.1622217},
year = {2009}
}
@book{Aigner2011a,
author = {Aigner, W. and Miksch, S. and Schumann, H. and Tominski, C.},
title = {{Visualization of Time-Oriented Data}},
doi = {10.1007/978-0-85729-079-3},
pages = {286},
publisher = {Springer},
url = {http://www.timeviz.net/},
year = {2011}
}
@article{Liu2013,
title = {{imMens: Real-time visual querying of big data}},
author = {Liu, Z. and Jiang, B. and Heer, J.},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
volume = {32},
year = {2013}
}
@inproceedings{Zuk2007,
author = {Zuk, T. and Carpendale, S.},
title = {{Visualization of uncertainty and reasoning}},
booktitle = {Proc. Intl. Symp. Smart Graphics},
doi = {10.1007/978-3-540-73214-3\_15},
pages = {164--177},
publisher = {Springer-Verlag},
year = {2007}
}
@article{Streit2008,
author = {Streit, A. and Pham, B. and Brown, R.},
doi = {10.1109/TVCG.2007.70426},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {61--72},
title = {{A spreadsheet approach to facilitate visualization of uncertainty in information}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17993702},
volume = {14},
year = {2008}
}
@inproceedings{Olston2002,
author = {Olston, C. and Mackinlay, J. D.},
title = {{Visualizing data with bounded uncertainty}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2002.1173145},
pages = {37--40},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1173145},
volume = {2002},
year = {2002}
}
@article{Booshehrian2012,
author = {Booshehrian, M. and M\"{o}ller, T. and Peterman, R. M. and Munzner, T.},
doi = {10.1111/j.1467-8659.2012.03116.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {1235--1244},
title = {{Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity In fisheries management decision making}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03116.x},
volume = {31},
year = {2012}
}
@article{Potter2010a,
author = {Potter, K. and Kniss, J. and Riesenfeld, R. and Johnson, C.R.},
doi = {10.1111/j.1467-8659.2009.01677.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {823--832},
title = {{Visualizing summary statistics and uncertainty}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2009.01677.x},
volume = {29},
year = {2010}
}
@article{Wickham,
author = {Wickham, H. and Stryjewski, L.},
journal = {The American Statistician},
pages = {1--17},
title = {40 years of boxplots}
}
@article{Robertson2009,
author = {Robertson, G. G. and Czerwinski, ./ and Fisher, D. and Lee, B.},
doi = {10.1518/155723409X448017},
journal = {Reviews of Human Factors and Ergonomics},
number = {1},
pages = {41--81},
title = {{Selected human factors issues in information visualization}},
url = {http://rev.sagepub.com/lookup/doi/10.1518/155723409X448017},
volume = {5},
year = {2009}
}
@article{Ma2012a,
author = {Ma, K. L. and Liao, I. and Frazier, J. and Hauser, H. and Kostis, H. N.},
doi = {10.1109/MCG.2012.24},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {1},
pages = {12--19},
title = {{Scientific storytelling using visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6111347},
volume = {32},
year = {2012}
}
@inproceedings{Perer2012a,
author = {Perer, A. and Sun, J.},
title = {{MatrixFlow: Temporal network visual analytics to track symptom evolution during disease progression}},
booktitle = {Proc. American Medical Informatics Association Annual Symposium (AMIA)},
pages = {1--10},
url = {http://www.research.ibm.com/healthcare/papers/AMIA2012MatrixFlow.pdf},
year = {2012}
}
@article{Agrawala2011,
author = {Agrawala, M. and Li, W. and Berthouzoz, F.},
doi = {10.1145/1924421.1924439},
issn = {00010782},
journal = {Communications of the ACM},
number = {4},
pages = {60},
title = {{Design principles for visual communication}},
url = {http://portal.acm.org/citation.cfm?doid=1924421.1924439},
volume = {54},
year = {2011}
}
@article{Toker2012,
author = {Toker, D. and Conati, C. and Carenini, G. and Haraty, M.},
journal = {User Modeling, Adaptation, and Personalization: Lecture Notes in Computer Science},
pages = {274--285},
title = {{Towards adaptive information visualization: On the influence of user characteristics}},
url = {http://www.springerlink.com/index/A452464U60622426.pdf},
volume = {7379},
year = {2012}
}
@article{Hollan2000,
author = {Hollan, J. and Hutchins, E. and Kirsh, D.},
doi = {10.1145/353485.353487},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {2},
pages = {174--196},
title = {{Distributed cognition: toward a new foundation for human-computer interaction research}},
url = {http://portal.acm.org/citation.cfm?doid=353485.353487},
volume = {7},
year = {2000}
}
@inproceedings{Fekete2011,
author = {Fekete, J. D. and Hemery, P. L. and Baudel, T. and Wood, J.},
title = {{Obvious: A meta-toolkit to encapsulate information visualization toolkits 場oolkit to bind them all}}
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2011.6102446},
pages = {91--100},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6102446},
year = {2011}
}
@article{Fisher1993,
author = {Fisher, P. and Dykes, J. and Wood, J.},
journal = {Cartographic Journal},
number = {2},
pages = {136--142},
title = {{Map design and visualization}},
volume = {30},
year = {1993}
}
@inproceedings{Carenini2004,
author = {Carenini, G. and Lloyd, J.},
title = {{ValueCharts: Analyzing linear models expressing preferences and valuations}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {150--157},
year = {2004}
}
@article{Mackinlay2007,
author = {Mackinlay, J. D. and Hanrahan, P. and Stolte, C.},
doi = {10.1109/TVCG.2007.70594},
issn = {1077-2626},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {1137--1144},
title = {{Show me: Automatic presentation for visual analysis}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968057},
volume = {13},
year = {2007}
}
@article{North2000,
author = {North, C. and Shneiderman, B.},
journal = {Intl. J. Human-Computer Studies},
number = {5},
pages = {715--739},
title = {{Snap-together visualization: Can users connect and operate coordinated visualizations?}},
volume = {53},
year = {2000}
}
@article{Upson1989,
author = {Upson, C. and Faulhaber, T. and Laidlaw, D. and Schlegel, D. and Gurwitz, R. and van Dam, A.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {4},
pages = {30--42},
title = {{The application visualization system: A computational environment for scientific visualization}},
volume = {9},
year = {1989}
}
@inproceedings{Kosara2003,
author = {Kosara, R. and Hauser, H. and Gresh, D.},
title = {{An interaction view on information visualization}},
booktitle = {Proc. Eurographics},
pages = {123--137},
year = {2003}
}
@article{Becker1987,
author = {Becker, R. A. and Cleveland, W. S. and Wilks, A. R.},
journal = {Statistical Science},
pages = {355--383},
title = {{Dynamic graphics for data analysis}},
volume = {2},
year = {1987}
}
@article{Wattenberg2006,
author = {Wattenberg, M. and Kriss, J.},
doi = {10.1109/TVCG.2006.65},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {4},
pages = {549--557},
title = {{Designing for social data analysis}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/16805263},
volume = {12},
year = {2006}
}
@inproceedings{Card1994,
author = {Card, S. K. and Pirolli, P. and Mackinlay, J. D.},
title = {{The cost-of-knowledge characteristic function: Display evolution for direct-walk dynamic interaction visualization}},
booktitle = {Proc. ACM SIGCHI Conf. on Human Factors in Computing Systems (CHI)},
pages = {238--244},
year = {1994}
}
@article{Heer2010a,
author = {Heer, J. and Bostock, M. and Ogievetsky, V.},
doi = {10.1145/1743546},
journal = {Communications of the ACM},
number = {6},
pages = {59--67},
title = {{A tour through the Visualization zoo}},
volume = {53},
year = {2010}
}
@article{Forbes2010,
author = {Forbes, A. G. and H⥲, T. and Legrady, G.},
doi = {10.1109/TVCG.2010.126},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1164--1171},
title = {{Behaviorism: A framework for dynamic data visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975155},
volume = {16},
year = {2010}
}
@article{Bostock2009,
author = {Bostock, M. and Heer, J.},
doi = {10.1109/TVCG.2009.174},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1121--1128},
title = {{Protovis: a graphical toolkit for visualization.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834180},
volume = {15},
year = {2009}
}
@article{Ziemkiewicz2008,
author = {Ziemkiewicz, C. and Kosara, R.},
doi = {10.1109/TVCG.2008.171},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1269--76},
title = {{The shaping of information by visual metaphors}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988973},
volume = {14},
year = {2008}
}
@article{Wickham2014,
author = {Wickham, H.},
title = {{Tidy data}},
journal = {J. Statistical Software},
pages = {1--23},
year = {2014}
}
@article{Emerson2011,
author = {Emerson, J. W. and Green, W. A. and Schloerke, B and Crowley, J. and Cook, D. and Hofmann, H. and Wickham, H.},
pages = {1--16},
title = {{The Generalized Pairs Plot}},
year = {2011}
}
@inproceedings{Hill1991,
author = {Hill, W. C. and Hollan, J. D.},
title = {{Deixis and the future of visualization excellence}},
booktitle = {Proc. IEEE Visualization},
doi = {10.1109/VISUAL.1991.175820},
pages = {314--320},
publisher = {IEEE Comput. Soc. Press},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=175820},
year = {1991}
}
@inproceedings{Nagel2012,
author = {Nagel, T. and Duval, E. and Moere, A. V.},
title = {{Interactive Exploration of Geospatial Network Visualization}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {557--572},
year = {2012}
}
@inproceedings{Jansen2012,
author = {Jansen, Y. and Dragicevic, P. and Fekete, J. D.},
title = {{Tangible remote controllers for wall-size displays}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208691},
pages = {2865},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208691},
year = {2012}
}
@inproceedings{Diakopoulos2012,
author = {Diakopoulos, N. and {De Choudhury}, M. and Naaman, M.},
title = {{Finding and assessing social media information sources in the context of journalism}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208409},
pages = {2451},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208409},
year = {2012}
}
@inproceedings{Dunne2012a,
author = {Dunne, C. and Riche, N. H. and Lee, B. and Metoyer, R. A. and Robertson, G. G.},
title = {{GraphTrail: Analyzing Large Multivariate, Heterogeneous Networks while Supporting Exploration History}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1663--1672},
year = {2012}
}
@inproceedings{Thudt2012,
author = {Thudt, A. and Hinrichs, U. and Carpendale, S.},
title = {{The Bohemian Bookshelf: Supporting Serendipitous Book Discoveries through Information Visualization}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1461--1470},
year = {2012}
}
@inproceedings{Kong2012,
author = {Kong, N. and Grossman, T. and Hartmann, B. and Fitzmaurice, G. and Agrawala, M.},
title = {{Delta: A Tool For Representing and Comparing Workflows}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1027--1036},
year = {2012}
}
@inproceedings{Faste2012,
author = {Faste, H. and Lin, H.},
title = {{The untapped promise of digital mind maps}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208548},
pages = {1017},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208548},
year = {2012}
}
@inproceedings{Pierce2012,
author = {Pierce, J.},
title = {{Undesigning Technology: Considering the Negation of Design by Design}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {957--966},
year = {2012}
}
@article{Yi2005,
author = {Yi, J. S. and Melton, R. and Stasko, J. and Jacko, J. A.},
doi = {10.1057/palgrave.ivs.9500099},
journal = {Information Visualization},
number = {4},
pages = {239--256},
title = {{Dust \& Magnet: multivariate information visualization using a magnet metaphor}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500099},
volume = {4},
year = {2005}
}
@inproceedings{Saraiya2005c,
author = {Saraiya, P. and Lee, P. and North, C.},
title = {{Visualization of graphs with associated timeseries data}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/INFVIS.2005.1532151},
pages = {225--232},
publisher = {Ieee},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532151},
year = {2005}
}
@article{Barsky2008,
author = {Barsky, A. and Munzner, T. and Gardy, J. and Kincaid, R.},
doi = {10.1109/TVCG.2008.117},
journal = {IEEE trans. visualization and computer graphics},
number = {6},
pages = {1253--1260},
title = {{Cerebral: visualizing multiple experimental conditions on a graph with biological context.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988971},
volume = {14},
year = {2008}
}
@incollection{Few2008,
author = {Few, S.},
title = {{What Ordinary People Need Most from Information Visualization Today}},
booktitle = {Perceptual Edge},
pages = {1--7},
url = {http://www.perceptualedge.com/articles/visual\_business\_intelligence/what\_people\_need\_from\_infovis.pdf},
year = {2008}
}
@techreport{Danziger2008,
author = {Danziger, M.},
pages = {1--89},
school = {Massechusetts Institute of Technology},
title = {{Information Visualization for the People}},
year = {2008}
}
@techreport{Ghahary2009,
author = {Ghahary, A.},
school = {SImon Fraser University},
title = {{Towards an Ideal in Informative Art}},
year = {2009}
}
@inproceedings{Luther2009,
author = {Luther, K. and Counts, S. and Stecher, KB and Hoff, A.},
title = {{Pathfinder: An Online Collaboration Environment for Citizen Scientists}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '09)},
pages = {239--248},
url = {http://dl.acm.org/citation.cfm?id=1518741},
year = {2009}
}
@article{Segel2010,
author = {Segel, E. and Heer, J.},
doi = {10.1109/TVCG.2010.179},
journal = {IEEE trans. visualization and computer graphics},
number = {6},
pages = {1139--1148},
title = {{Narrative visualization: telling stories with data.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20975152},
volume = {16},
year = {2010}
}
@article{Potter2010,
author = {Potter, K. and Kniss, J. and Riesenfeld, R. and Johnson, C. R.},
journal = {Computer Graphics Forum},
number = {3},
pages = {823--832},
publisher = {Wiley Online Library},
title = {{Visualizing summary statistics and uncertainty}},
url = {http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8659.2009.01677.x/full},
volume = {29},
year = {2010}
}
@article{Skeels2009,
author = {Skeels, M. and Lee, B. and Smith, G. and Robertson, G. G.},
doi = {10.1057/ivs.2009.1},
journal = {Information Visualization},
number = {1},
pages = {70--81},
title = {{Revealing uncertainty for information visualization}},
url = {http://www.palgrave-journals.com/doifinder/10.1057/ivs.2009.1},
volume = {9},
year = {2009}
}
@article{Gleicher2011,
author = {Gleicher, M. and Albers, D. and Walker, R. and Jusufi, I. and Hansen, C. D. and Roberts, J. C.},
doi = {10.1177/1473871611416549},
journal = {Information Visualization},
title = {{Visual comparison for information visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611416549},
year = {2011}
}
@article{Moere2011,
author = {Moere, A. V. and Purchase, H.},
doi = {10.1177/1473871611415996},
journal = {Information Visualization},
title = {{On the role of design in information visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415996},
year = {2011}
}
@article{Heer2006a,
author = {Heer, J. and Agrawala, M.},
doi = {10.1109/TVCG.2006.178},
journal = {IEEE transactions on visualization and computer graphics},
number = {5},
pages = {853--60},
title = {{Software design patterns for information visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080809},
volume = {12},
year = {2006}
}
@inproceedings{Dix2007,
author = {Dix, A.},
title = {{Designing for Appropriation}},
booktitle = {Proc. of the 21st BCS HCI Group Conference},
number = {September},
pages = {28--30},
volume = {2},
year = {2007}
}
@incollection{Gould1996,
author = {Gould, J. D.},
title = {{How to Design Usable Systems}},
booktitle = {Readings in human-computer interaction: Towards the Year 2000},
editor = {Baecker, R. M. and Grudin, J. and Buxton, W. and Greenberg, S.},
pages = {93--121},
publisher = {Morgan Kaufmann},
year = {1996}
}
@article{Maguire2001,
author = {Maguire, M.},
title = {{Methods to support human-centred design}},
doi = {10.1006/ijhc.2001.0503},
journal = {International Journal of Human-Computer Studies},
number = {4},
pages = {587--634},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581901905038},
volume = {55},
year = {2001}
}
@inproceedings{Dourish2006,
author = {Dourish, P.},
title = {{Implications for design}},
booktitle = {Proceedings of the SIGCHI conference on Human Factors in computing systems (CHI '06)},
doi = {10.1145/1124772.1124855},
pages = {541--550},
url = {http://portal.acm.org/citation.cfm?doid=1124772.1124855},
year = {2006}
}
@inproceedings{Zimmerman2007,
author = {Zimmerman, J. and Forlizzi, J. and Evenson, S. and Zimmerman, J. and Forlizzi, J. and Evenson, S.},
title = {{Research Through Design as a Method for Interaction Design Research in HCI design research in HCI}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '07)},
pages = {493--502},
year = {2007}
}
@article{Friedman2003a,
author = {Friedman, K.},
title = {{Theory construction in design research: criteria: approaches, and methods}},
doi = {10.1016/S0142-694X(03)00039-5},
journal = {Design Studies},
number = {6},
pages = {507--522},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0142694X03000395},
volume = {24},
year = {2003}
}
@inproceedings{Goodman2011,
author = {Goodman, E. and Wakkary, R.},
title = {{Understanding Interaction Design Practices}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '11)},
pages = {1061--1070},
year = {2011}
}
}}}
{{{
@techreport{Sundaramurthy,
author = {Sundaramurthy, S. C. and McHugh, J. and Ou, X. and Rajagopalan, S. R. and Wesch, M.},
title = {{An anthropological approach to studying CSIRTs}}
}
@article{Dittrich2007,
author = {Dittrich, Y. and R殫k欠K. and Eriksson, J. and Hansson, C. and Lindeberg, O.},
title = {{Cooperative method development}},
journal = {Empirical Software Engineering},
number = {3},
pages = {231--260},
volume = {13},
year = {2008}
}
@inproceedings{Baumer2011,
author = {Baumer, E. P. S. and Tomlinson, B.},
title = {{Comparing activity theory with distributed cognition for video analysis: Beyond Kicking the tires}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {133--142},
year = {2011}
}
@article{Hornbaek2013,
author = {Hornbae k, K.},
doi = {10.1561/1100000043},
journal = {Some whys and hows of experiments in human-computer interaction},
number = {4},
pages = {299--373},
title = {{Some Whys and Hows of Experiments in Humanﭰuter Interaction}},
url = {http://www.nowpublishers.com/articles/foundations-and-trends-in-humancomputer-interaction/HCI-043},
volume = {5},
year = {2013}
}
@article{Mullins1991,
author = {Mullins, P. M. and Treu, S.},
journal = {SIGCHI Bulletin},
pages = {52--53},
title = {{Task protocol specification (TPS): a workstation independent specification for human-computer interaction}},
volume = {23},
year = {1991}
}
@article{Carter1986,
author = {Carter, J.},
journal = {Intl. J. Man Machine Studies},
pages = {195--292},
title = {{A taxonomy of user-oriented functions}},
volume = {24},
year = {1986}
}
@article{Gawron1989,
author = {Gawron, V. and Drury, C. G. and Czaja, S. J. and Wilkins, D. M},
journal = {Intl. J. Man Machine Studies},
pages = {643--672},
title = {{A taxonomy of independent human performance}},
volume = {31},
year = {1989}
}
@inproceedings{Quinn2011,
author = {Quinn, A. J. and Bederson, B. B.},
title = {{Human Computation: A Survey and Taxonomy of a Growing Field}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1403--1412},
year = {2011}
}
@inproceedings{Kulkarni2012,
author = {Kulkarni, A. and Can, M. and Hartmann, B.},
title = {{Collaboratively crowdsourcing workflows with Turkomatic}},
booktitle = {Proc. ACM Conf. Computer Supported Cooperative Work (CSCW)},
doi = {10.1145/2145204.2145354},
url = {http://dl.acm.org/citation.cfm?doid=2145204.2145354},
year = {2012}
}
@inproceedings{Kittur2011,
author = {Kittur, A. and Smus, B. and Kraut, R.},
title = {{CrowdForge: Crowdsourcing Complex Work}},
booktitle = {Proc. Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1801--1806},
year = {2011}
}
@inproceedings{Ross2010,
author = {Ross, J. and Irani, L. and Silberman, M. S. and Zaldivar, A. and Tomlinson, B.},
title = {{Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk}},
booktitle = {Proc. Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2863--2872},
year = {2010}
}
@article{Alonso2008,
author = {Alonso, Omar and Rose, Daniel E. and Stewart, Benjamin},
doi = {10.1145/1480506.1480508},
journal = {ACM SIGIR Forum},
month = nov,
number = {2},
pages = {9},
title = {{Crowdsourcing for relevance evaluation}},
url = {http://portal.acm.org/citation.cfm?doid=1480506.1480508},
volume = {42},
year = {2008}
}
@inproceedings{Mackay2007,
author = {Mackay, W. E. and Appert, C. and Beaudouin-Lafon, M.},
title = {{Touchstone: Exploratory Design of Experiments}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1425--1434},
year = {2007}
}
@article{Kohavi2007,
author = {Kohavi, R. and Henne, R. M.},
journal = {ACM KDD},
pages = {959--967},
title = {{Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the Hippo}},
year = {2007}
}
@inproceedings{Kittur2008,
title = {{Crowdsourcing user studies with Mechanical Turk}},
author = {Kittur, A. and Chi, E. H. and Suh, B.},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/1357054.1357127},
pages = {453},
publisher = {ACM},
url = {http://portal.acm.org/citation.cfm?doid=1357054.1357127},
year = {2008}
}
@article{Mason2009,
author = {Mason, W. A and Watts, D. J.},
journal = {ACM KDD-HCOMP},
number = {2},
pages = {100--108},
title = {{Financial Incentives and the ॲformance of Crowds �volume = {11},
year = {2009}
}
@article{Kraut2004,
author = {Kraut, R. and Olson, J. and Banaji, M. and Bruckman, A. and Cohen, J. and Couper, M.},
doi = {10.1037/0003-066X.59.2.105},
journal = {American Psychologist},
number = {2},
pages = {105--17},
title = {{Psychological research online: report of Board of Scientific Affairs' Advisory Group on the Conduct of Research on the Internet.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/14992637},
volume = {59},
year = {2004}
}
@inproceedings{Antin2012,
author = {Antin, J. and Shaw, A.},
title = {Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
year = {2012},
pages = {2925--2934},
url = {http://doi.acm.org/10.1145/2207676.2208699},
}
@article{Robertson2012,
author = {Robertson, J.},
doi = {10.1145/2160718.2160721},
journal = {Communications of the ACM},
number = {5},
pages = {6},
title = {{Likert-type scales, statistical methods, and effect sizes}},
url = {http://dl.acm.org/citation.cfm?doid=2160718.2160721},
volume = {55},
year = {2012}
}
@inproceedings{Dell2012,
author = {Dell, N. and Vaidyanathan, V. and Medhi, I. and Cutrell, E. and Thies, W.},
title = {{"Yours is Better" Participant Response Bias in HCI}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1321--1330},
year = {2012}
}
@inproceedings{Friess2012,
author = {Friess, E.},
title = {{Personas and Decision Making in the Design Process: An Ethnographic Case Study}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1209--1218},
year = {2012}
}
@inproceedings{Rode2012,
author = {Rode, J. A. and Blythe, M. and Nardi, B.},
title = {{Qualitative Research in HCI}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2803--2806},
year = {2012}
}
@inproceedings{Swearngin2012,
author = {Swearngin, A. and Cohen, M. and John, B. and Bellamy, R.},
title = {{Easing the generation of predictive human performance models from legacy systems}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208415},
pages = {2489},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208415},
year = {2012}
}
@inproceedings{Karapanos2012,
author = {Karapanos, E. and Jain, J. and Hassenzahl, M.},
title = {{Theories, Methods and Case Studies of Longitudinal HCI Research}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2727--2730},
year = {2012}
}
@inproceedings{Obrist2012,
author = {Obrist, M. and V\"{a}\"{a}n\"{a}nen-Vainio-Mattila, K. and Roto, V. and Vermeeren, A. and Law, E. L. and Buie, E.},
title = {{Theories behind UX Research and How They Are Used in Practice}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2751--2754},
year = {2012}
}
@inproceedings{Gomez2012,
author = {Gomez, S. and Laidlaw, D.},
title = {{Modeling task performance for a crowd of users from interaction histories}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208412},
pages = {2465},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208412},
year = {2012}
}
@inproceedings{Teo2012,
author = {Teo, L. H. and John, B. E. and Blackmon, M. H.},
title = {{CogTool-Explorer: A Model of Goal-Directed User Exploration that Considers Information Layout}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2479--2488},
year = {2012}
}
@inproceedings{Abeele2012,
author = {Abeele, V. V. and Hauters, E. and Zaman, B.},
title = {{Increasing the Reliability and Validity of Quantitative Laddering Data with LadderUX}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {2057--2062},
year = {2012}
}
@inproceedings{Kusunoki2012,
author = {Kusunoki, D.},
title = {{Applying Participatory Design Theory to Designing Evaluation Methods}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1895--1900},
year = {2012}
}
@inproceedings{Obrist2012a,
author = {Obrist, M. and V\"{a}\"{a}n\"{a}nen-Vainio-Mattila, K. and Roto, V. and Law, E. L. and Vermeeren, A. and Kuutti, K.},
title = {{In Search of Theoretical Foundations for UX Research and Practice}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
year = {2012}
}
@inproceedings{Kaptein2012,
author = {Kaptein, M. and Robertson, J.},
title = {{Rethinking statistical analysis methods for CHI}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/2207676.2208557},
pages = {1105},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208557},
year = {2012}
}
@inproceedings{Jain2012,
author = {Jain, J. and Boyce, S.},
title = {{Case Study: Longitudinal Comparative Analysis for Analyzing User Behavior}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {793--799},
year = {2012}
}
@inproceedings{Stitzky2012,
author = {Stitzky, A.},
title = {{Incorporating UCD Into the Software Development Process: A Case Study}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {469--483},
year = {2012}
}
@inproceedings{Lingel2012,
author = {Lingel, J.},
title = {{Ethics and Dilemmas of Online Ethnography}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {41--50},
year = {2012}
}
@inproceedings{Gill2012,
author = {Gill, Z.},
title = {{User-Driven Collaborative Intelligence Social Networks as Crowdsourcing Ecosystems}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {161--170},
year = {2012}
}
@inproceedings{Ko2012,
author = {Ko, A. J.},
title = {{Mining Whining in Support Forums with Frictionary}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {191--200},
year = {2012}
}
@inproceedings{Matavire2008,
author = {Matavire, R. and Brown, I.},
title = {{Investigating the use of grounded theory in information systems research}},
booktitle = {Proc. ACM SAICSIT},
pages = {139--147},
year = {2008}
}
@article{Urquhart2001,
author = {Urquhart, C.},
journal = {Qualitative research in IS: issues and trends},
title = {{An encounter with grounded theory: Tackling the practical and philosophical issues}},
year = {2001}
}
@incollection{Adams2008,
author = {Adams, A. and Lunt, P. and Cairns, P.},
title = {{A qualititative approach to HCI research}},
booktitle = {Research methods for Human-Computer Interaction},
editor = {Cairns, P. and Cox, A.},
pages = {138--157},
publisher = {Cambridge University Press},
year = {2008}
}
@incollection{Henwood2003,
author = {Henwood, K. and Pidgeon, N.},
title = {{Grounded theory in psychological research}},
booktitle = {Qualitative research in psychology: Expanding perspectives in methodology and design},
editor = {Camic, P. and Rhodes, J. and Yardley, L.},
pages = {131--155},
publisher = {APA},
year = {2003}
}
@article{Hartson2001,
author = {Hartson, H. R. and Andre, T. S.},
journal = {International Journal of Human-Computer Interaction (IJHCI)},
number = {4},
pages = {373--410},
title = {{Criteria For Evaluating Usability Evaluation Methods}},
volume = {13},
year = {2001}
}
@inproceedings{Brush2004,
title = {{A comparison of synchronous remote and local usability studies for an expert interface}},
author = {Brush, A. J. and Ames, M. and Davis, J.},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
doi = {10.1145/985921.986018},
pages = {1179--1182},
publisher = {ACM Press},
url = {http://portal.acm.org/citation.cfm?doid=985921.986018},
year = {2004}
}
@inproceedings{Steves2001,
title = {{A comparison of usage evaluation and inspection methods for assessing groupware usability}},
author = {Steves, M. P. and Morse, E. and Gutwin, C. and Greenberg, S.},
booktitle = {Proc. Intl. ACM SIGGROUP Conf. Supporting Group Work (GROUP)},
doi = {10.1145/500300.500306},
pages = {125},
publisher = {ACM Press},
url = {http://portal.acm.org/citation.cfm?doid=500286.500306},
year = {2001}
}
@inproceedings{Doubleday1997,
title = {{A comparison of usability techniques for evaluating design}},
author = {Doubleday, A. and Ryan, M. and Springett, M. and Sutcliffe, A.},
booktitle = {Proc. Conf. Designing Interactive Systems (DIS)},
doi = {10.1145/263552.263583},
pages = {101--110},
publisher = {ACM Press},
url = {http://portal.acm.org/citation.cfm?doid=263552.263583},
year = {1997}
}
@article{Jeffries1992,
author = {Jeffries, R. and Desurviret, H.},
journal = {SIGCHI Bulletin},
number = {4},
pages = {39--41},
title = {{Usability Testing vs. Heuristic Evaluation: Was there a contest?}},
volume = {24},
year = {1992}
}
@article{John1997,
author = {John, B. and Marks, S.},
journal = {Behaviour and Information Technology},
number = {4/5},
pages = {188--202},
title = {{Tracking the effectiveness of usability evaluation methods}},
url = {http://www.tandfonline.com/doi/abs/10.1080/014492997119789},
volume = {16},
year = {1997}
}
@article{Gaver1999,
author = {Gaver, B. and Dunne, T. and Pacenti, E.},
journal = {interactions},
number = {1},
pages = {21--29},
title = {{Design: cultural probes}},
url = {http://dl.acm.org/citation.cfm?id=291235},
volume = {6},
year = {1999}
}
@article{Grammel2010,
author = {Grammel, L. and Tory, M. and Storey, M.},
journal = {IEEE Trans. Visualization and Computer Graphics},
number = {6},
pages = {943--952},
title = {{How information visualization novices construct visualizations}},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=5613431},
volume = {16},
year = {2010}
}
@incollection{Heer2008,
author = {Heer, J. and van Ham, F and Carpendale, S. and Weaver, C.},
title = {{Creation and Collaboration: Engaging New Audiences for Information Visualization}},
booktitle = {Information Visualization: Human-centered issues and perspectives},
pages = {92--133},
publisher = {Springer-Verlag},
url = {http://www.springerlink.com/index/15jg8jt161k51kq7.pdf},
year = {2008}
}
@inproceedings{Graham2007,
author = {Graham, C. and Rouncefield, M. and Gibbs, M. and Vetere, F. and Cheverst, K.},
title = {{How Probes Work}},
booktitle = {Proc. Austrialian Conf. Computer-Human Interaction (OZCHI)},
pages = {29--37},
year = {2007}
}
@article{Locke2002,
author = {Locke, E. A. and Latham, G. P.},
doi = {10.1037//0003-066X.57.9.705},
journal = {American Psychologist},
number = {9},
pages = {705--717},
title = {{Building a practically useful theory of goal setting and task motivation}},
url = {http://doi.apa.org/getdoi.cfm?doi=10.1037/0003-066X.57.9.705},
volume = {57},
year = {2002}
}
@article{Davis1989,
author = {Davis, F. D. and Bagozzi, R. P. and Warshaw, P. R.},
journal = {Management science},
number = {8},
pages = {982--1003},
title = {{User acceptance of computer technology: a comparison of two theoretical models}},
url = {http://www.jstor.org/stable/10.2307/2632151},
volume = {35},
year = {1989}
}
@article{Tierney1995,
author = {Tierney, W. G.},
doi = {10.1177/107780049500100401},
journal = {Qualitative Inquiry},
number = {4},
pages = {379--390},
title = {{(Re)Presentation and Voice}},
url = {http://qix.sagepub.com/cgi/doi/10.1177/107780049500100401},
volume = {1},
year = {1995}
}
@article{Eisner1981,
author = {Eisner, E. W.},
doi = {10.3102/0013189X010004005},
journal = {Educational Researcher},
number = {4},
pages = {5--9},
title = {{On the Differences Between Scientific and Artistic Approaches to Qualitative Research}},
url = {http://edr.sagepub.com/cgi/doi/10.3102/0013189X010004005},
volume = {10},
year = {1981}
}
@article{Chenail1995,
author = {Chenail, R. J.},
journal = {The Qualitative Report},
number = {3},
title = {{Presenting Qualitative Data}},
volume = {2},
year = {1995}
}
@article{Jang1994,
author = {Jang, H. Y. and Barnett, G. A.},
journal = {Bulletin de Methodologie Sociologique},
pages = {31--59},
title = {{Cultural diffeerences in organizational communication: A semantic network analysis}},
url = {http://bms.sagepub.com/content/44/1/31.short},
volume = {44},
year = {1994}
}
@article{Znaniecki1939,
author = {Znaniecki, F.},
journal = {American Journal of Sociology},
pages = {799--811},
publisher = {JSTOR},
title = {{Social Groups as Products of Participating Individuals}},
url = {http://www.jstor.org/stable/10.2307/2769412},
year = {1939}
}
@article{Robinson1951,
author = {Robinson, W. S.},
journal = {American Sociological Review},
number = {6},
pages = {812--818},
publisher = {JSTOR},
title = {{The logical structure of analytic induction}},
url = {http://www.jstor.org/stable/10.2307/2087508},
volume = {16},
year = {1951}
}
@inproceedings{Shaw2011,
author = {Shaw, A. D. and Horton, J. J. and Chen, D. L.},
booktitle = {Proceedings of the ACM conference on Computer supported cooperative work (CSCW '11)},
pages = {275--284},
publisher = {ACM},
title = {{Designing Incentives for Inexpert Human Raters}},
url = {http://dl.acm.org/citation.cfm?id=1958865},
year = {2011}
}
@article{Hayes2011,
author = {Hayes, Gillian R.},
doi = {10.1145/1993060.1993065},
journal = {ACM Transactions on Computer-Human Interaction},
number = {3},
pages = {1--20},
title = {{The relationship of action research to human-computer interaction}},
url = {http://dl.acm.org/citation.cfm?doid=1993060.1993065},
volume = {18},
year = {2011}
}
@inproceedings{Millen2000a,
author = {Millen, D. R.},
title = {{Rapid Ethnography: Time Deepening Strategies for HCI Field Research}},
booktitle = {Proc. of ACM DIS '00},
pages = {280--286},
year = {2000}
}
@book{Corbin2008,
author = {Corbin, J. and Strauss, A.},
edition = {3rd Edition},
pages = {1--379},
publisher = {Sage Publications, Inc.},
title = {{Basics of Qualitative Research}},
year = {2008}
}
@inproceedings{Bargas-Avila2011,
author = {Bargas-Avila, J. A. and Hornb\ae k, K.},
title = {{Old Wine in New Bottles or Novel Challenges? A Critical Analysis of Empirical Studies of User Experience}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '11)},
pages = {2689--2698},
year = {2011}
}
@inproceedings{Sengers2006,
author = {Sengers, P. and Gaver, B.},
title = {{Staying Open to Interpretation: Engaging Multiple Meanings in Design and Evaluation}},
booktitle = {Proc. DIS '06},
pages = {99--108},
year = {2006}
}
@article{Zhai2003,
author = {Zhai, S.},
title = {{Evaluation is the worst form of HCI research except all those other forms that have been tried}},
journal = {CHI Place},
pages = {1--3},
url = {http://www.shuminzhai.com/papers/EvaluationDemocracy.htm},
year = {2003}
}
@article{Lieberman2003,
author = {Lieberman, H.},
title = {{The Tyranny of Evaluation}},
journal = {ACM CHI Fringe},
url = {http://web.media.mit.edu/~lieber/Misc/Tyranny-Evaluation.html},
year = {2003}
}
@inproceedings{Kaye2007,
author = {Kaye, J. and Sengers, P.},
title = {{The Evolution of Evaluation}},
booktitle = {Proc. Alt.CHI '07},
url = {http://www.viktoria.se/altchi/submissions/submission\_jofish\_0.pdf},
year = {2007}
}
@inproceedings{OlsenJr2007,
author = {{Olsen Jr}, D.R.},
title = {{Evaluating user interface systems research}},
booktitle = {Proc. of the 20th annual ACM symposium on User interface software and technology (UIST '07)},
pages = {251--258},
url = {http://portal.acm.org/citation.cfm?id=1294256},
year = {2007}
}
@inproceedings{Dicks2002,
author = {Dicks, R. S.},
title = {{Mis-Usability: On the Uses and Misuses of Usability Testing}},
booktitle = {Proc. SIGDOC '02},
pages = {26--30},
year = {2002}
}
@inproceedings{Tohidi2006,
author = {Tohidi, M. and Buxton, W. and Baecker, R. and Sellen, A.},
title = {{User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback}},
booktitle = {Proc. of the 4th Nordic conference on Human-computer interaction: changing roles (NordiCHI '06)},
pages = {105--114},
url = {http://portal.acm.org/citation.cfm?id=1182487},
year = {2006}
}
@inproceedings{Tohidi2006a,
author = {Tohidi, M. and Buxton, W. and Baecker, R. and Sellen, A.},
title = {{Getting the Right Design and the Design Right: Testing Many Is Better Than One}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '06)},
pages = {1243--1252},
year = {2006}
}
@inproceedings{Barkhuus2007,
author = {Barkhuus, L. and Rode, J. A.},
title = {{From Mice to Men: 24 years of Evaluation in CHI}},
booktitle = {Extended abstracts of the 2007 conference on Human factors and computing systems (CHI '07)},
pages = {1--16},
year = {2007}
}
@article{Hilbert2000,
author = {Hilbert, D. M. and Redmiles, D. F.},
title = {{Extracting usability information from user interface events}},
journal = {ACM Computing Surveys},
number = {4},
pages = {384--421},
publisher = {ACM},
url = {http://portal.acm.org/citation.cfm?id=371593},
volume = {32},
year = {2000}
}
@article{Ivory2001,
author = {Ivory, M. Y. and Hearst, M.. A.},
title = {{The state of the art in automating usability evaluation of user interfaces}},
doi = {10.1145/503112.503114},
journal = {ACM Computing Surveys},
number = {4},
pages = {470--516},
url = {http://portal.acm.org/citation.cfm?doid=503112.503114},
volume = {33},
year = {2001}
}
@inproceedings{Nielsen1990,
author = {Nielsen, J. and Molich, R.},
title = {{Heuristic Evaluation of User Interfaces}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '90)},
pages = {249--256},
publisher = {ACM},
year = {1990}
}
@inproceedings{Jeffries1991,
author = {Jeffries, R. and Miller, J. R. and Wharton, C. and Uyeda, K.},
title = {{User interface evaluation in the real world: a comparison of four techniques}},
booktitle = {Proce. of the SIGCHI conference on Human factors in computing systems (CHI '91)},
number = {c},
pages = {119--124},
url = {http://portal.acm.org/citation.cfm?id=108862},
volume = {91},
year = {1991}
}
@article{Kensing1998,
author = {Kensing, F. and Simonsen, J. and Bodker, K.},
doi = {10.1207/s15327051hci1302\_3},
journal = {~Human-Computer Interaction},
number = {2},
pages = {167--198},
title = {{MUST: A Method for Participatory Design}},
url = {http://www.informaworld.com/openurl?genre=article\&doi=10.1207/s15327051hci1302\_3\&magic=crossref||D404A21C5BB053405B1A640AFFD44AE3},
volume = {13},
year = {1998}
}
@article{Duncan2004,
author = {Duncan, M.},
title = {{Autoethnography: Critical appreciation of an emerging art}},
journal = {International Journal of Qualitative Methods},
number = {4},
pages = {28--39},
volume = {3},
year = {2004}
}
@article{Gerring2004,
author = {Gerring, J.},
title = {{What Is a Case Study and What Is It Good for?}},
doi = {10.1017/S0003055404001182},
journal = {American Political Science Review},
number = {2},
pages = {341--354},
url = {http://www.journals.cambridge.org/abstract\_S0003055404001182},
volume = {98},
year = {2004}
}
@article{Greene1989,
author = {Greene, J. C. and Caracelli, V. J. and Graham, W. F.},
title = {{Toward a Conceptual Framework for Mixed-Method Evaluation Designs}},
doi = {10.3102/01623737011003255},
journal = {Educational Evaluation and Policy Analysis},
number = {3},
pages = {255--274},
url = {http://epa.sagepub.com/cgi/doi/10.3102/01623737011003255},
volume = {11},
year = {1989}
}
@article{Carroll2000,
author = {Carroll, J.},
title = {{Five reasons for scenario-based design}},
doi = {10.1016/S0953-5438(00)00023-0},
journal = {Interacting with Computers},
number = {1},
pages = {43--60},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0953543800000230},
volume = {13},
year = {2000}
}
@article{Janlert2010,
author = {Janlert, L. E. and Stolterman, E.},
title = {{Complex Interaction}},
doi = {10.1145/1746259.1746262},
journal = {ACM Transactions on Computer-Human Interaction},
number = {2},
pages = {1--32},
url = {http://portal.acm.org/citation.cfm?doid=1746259.1746262},
volume = {17},
year = {2010}
}
@inproceedings{Rode2011,
author = {Rode, J. A.},
title = {{Reflexivity in digital anthropology}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {123--132},
year = {2011}
}
@article{Rogers2004,
author = {Rogers, Y.},
title = {{New theoretical approaches for human-computer interaction}},
journal = {Annual Review of Information Science and Technology},
number = {1},
pages = {87--143},
volume = {38},
year = {2004}
}
}}}
{{{
@article{Bush1945,
author = {Bush, V.},
title = {{As We May Think}},
journal = {The Atlantic Monthly},
pages = {35--47},
publisher = {MIT Press, Cambridge},
year = {1945}
}
}}}
{{{
@inproceedings{Chen2011,
author = {Chen, Y. and Alsakran, J. and Barlowe, S. and Yang, J. and Zhao, Y.},
title = {{Supporting effective common ground construction in asynchronous collaborative visual analytics}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology (VAST)},
pages = {99--108},
year = {2011}
}
@inproceedings{Kleek2013,
author = {Kleek, M. v. and Smith, D. A. and Packer, H. S. and Skinner, J. and Shadbolt, N. R.},
title = {{Carp\'{e} data: Supporting serendipitous data integration in personal information management}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013}
}
@article{Nersessian2006,
author = {Nersessian, N. J.},
doi = {10.1177/0170840606061842},
journal = {Organization Studies},
number = {1},
pages = {125--145},
title = {{The cognitive-cultural systems of the research laboratory}},
url = {http://oss.sagepub.com/cgi/doi/10.1177/0170840606061842},
volume = {27},
year = {2006}
}
@incollection{Nersessian2008,
author = {Nersessian, N. J.},
title = {{Mental modeling in conceptual change}},
booktitle = {Handbook of Conceptual Change},
editor = {Vosniadou, S.},
pages = {1--78},
publisher = {Routledge},
year = {2008}
}
@inproceedings{Zhang1993,
author = {Zhang, J.},
title = {{The Interaction between perceptual and cognitive processes in a distributed problem solving task}},
booktitle = {Proc. Fall AAAI Symp. Games: Planning and Learning},
number = {1},
pages = {1--9},
year = {1993}
}
@inproceedings{Craft2005,
author = {Craft, B. and Cairns, P.},
title = {{Beyond guidelines: What can we learn from the Visual Information Seeking Mantra?}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/IV.2005.28},
pages = {110--118},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1509067},
year = {2005}
}
@inproceedings{Viegas2004,
author = {Vi젆. B. and Boyd, D. and Nguyen, D. H. and Potter, J. and Donath, J.},
title = {{Digital artifacts for remembering and storytelling: posthistory and social network fragments}},
booktitle = {Proc. IEEE Annual Hawaii International Conference on System Sciences (HICSS)},
doi = {10.1109/HICSS.2004.1265287},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1265287},
volume = {00},
year = {2004}
}
@article{Zhang1994,
author = {Zhang, J. and Norman, D. A.},
doi = {10.1016/0364-0213(94)90021-3},
journal = {Cognitive Science},
number = {1},
pages = {87--122},
title = {{Representations in distributed cognitive tasks}},
url = {http://doi.wiley.com/10.1016/0364-0213(94)90021-3},
volume = {18},
year = {1994}
}
@inproceedings{Dadzie2009,
author = {Dadzie, A. S. and Petrelli, D.},
title = {{Visual knowledge exploration and discovery from different points of view}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2009.5333438},
pages = {227--228},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5333438},
year = {2009}
}
@article{Bhatt2008,
author = {Bhatt, G. D.},
journal = {Knowledge Management},
number = {1},
pages = {15--26},
title = {{Organizing knowledge in the knowledge development cycle}},
volume = {4},
year = {2008}
}
@article{Bates1989,
author = {Bates, M. J.},
journal = {Online Review},
number = {13},
pages = {407--424},
title = {{The design of browsing and berrypicking techniques for the online search interface}},
year = {1989}
}
@article{James2006,
author = {James, J. J. and Cook, K. A.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
pages = {10--13},
title = {{A visual analytics agenda}},
year = {2006}
}
@article{Andrienko2007,
author = {Andrienko, G. and Andrienko, N. and Jankowski, P. and Keim, D. and Kraak, M. J. and MacEachren, A. M. and Wrobel, S.},
journal = {Intl. J. Geographical Information Science},
number = {8},
pages = {839--857},
title = {{Geovisual analytics for spatial decision support: Setting the research agenda}},
volume = {21},
year = {2007}
}
@article{Scholtz2004,
author = {Scholtz, J. and Morse, E. and Hewett, T.},
journal = {Human Performance, Situation Awareness, and Automation},
title = {{In depth observational studies of professional intelligence analysts}},
year = {2004}
}
@inproceedings{Patterson2001,
author = {Patterson, E. S. and Woods, D. D. and Tinapple, D. and Roth, E. M.},
title = {{Using cognitive task analysis (CTA) to seed design concepts for intelligence analysts under data overload}},
booktitle = {Proc. Human Factors and Ergonomics Soc.},
doi = {10.1177/154193120104500437},
number = {4},
pages = {439--443},
url = {http://pro.sagepub.com/lookup/doi/10.1177/154193120104500437},
volume = {45},
year = {2001}
}
@article{Eccles2008,
author = {Eccles, R. and Kapler, T. and Harper, R. and Wright, W.},
doi = {10.1057/palgrave.ivs.9500173},
journal = {Information Visualization},
number = {1},
pages = {3--17},
title = {{Stories in GeoTime}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500173},
volume = {7},
year = {2008}
}
@inproceedings{Hampson2005,
author = {Hampson, E. and Cowley, P.},
title = {{Instrumenting the intelligence analysis process}},
booktitle = {Proc. Intl. Conf. Intelligence Analysis},
year = {2005}
}
@inproceedings{Gotz2006b,
author = {Gotz, D. and Zhou, M. X.},
title = {{A study of information gathering and result processing in intelligence analysis}},
booktitle = {Proc. ACM WS. IUI for Intelligence Analysis},
year = {2006}
}
@inproceedings{Bavoil2005,
author = {Bavoil, L. and Callahan, S. P. and Crossno, P. J. and Freire, J. and Scheidegger, C. E. and Silva, T. and Vo, H. T.},
title = {{VisTrails: Enabling interactive multiple-view visualizations}},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {135--142},
year = {2005}
}
@inproceedings{Morton2012,
author = {Morton, K. and Bunker, R. and Mackinlay, J. D. and Morton, R. and Stolte, C.},
title = {{Dynamic workload driven data integration in tableau}},
booktitle = {Proc. ACM Intl. Conf. Management of Data (SIGMOD)},
doi = {10.1145/2213836.2213961},
pages = {807--815},
url = {http://dl.acm.org/citation.cfm?doid=2213836.2213961},
year = {2012}
}
@book{Case2008,
author = {Case, D. O.},
publisher = {Emerald Publishing Ltd.},
title = {{Looking for Information}},
year = {2008}
}
@article{Foster2003,
author = {Foster, A. and Ford, N.},
doi = {10.1108/00220410310472518},
journal = {Documentation},
number = {3},
pages = {321--340},
title = {{Serendipity and information seeking: An empirical study}},
url = {http://www.emeraldinsight.com/10.1108/00220410310472518},
volume = {59},
year = {2003}
}
@article{Wilson1999,
author = {Wilson, T. D.},
journal = {Documentation},
number = {3},
pages = {249--270},
title = {{Models of information behaviour research}},
volume = {55},
year = {1999}
}
@article{Ellis1997,
author = {Ellis, D. and Haugan, M.},
doi = {10.1108/EUM0000000007204},
journal = {Documentation},
number = {4},
pages = {384--403},
title = {{Modelling the information seeking patterns of engineers and research scientists in an industrial environment}},
url = {http://www.emeraldinsight.com/10.1108/EUM0000000007204},
volume = {53},
year = {1997}
}
@article{Wilson2000,
author = {Wilson, T. D.},
journal = {Informing Science},
number = {2},
pages = {49--56},
title = {{Human information behavior}},
volume = {3},
year = {2000}
}
@article{Li2009,
author = {Li, Y.},
doi = {10.1002/asi},
journal = {J. Am Soc. Information Science and Technology},
number = {2},
pages = {275--291},
title = {{Exploring the relationships between work task and search task in information search}},
volume = {60},
year = {2009}
}
@inproceedings{Toms1999,
author = {Toms, E. G.},
title = {{Serendipitous information retrieval}},
booktitle = {Proc. WS. Information Seeking, Searching and Querying in Digital Libraries},
number = {1968},
pages = {1--4},
year = {1999}
}
@article{Bystrom1995,
author = {Bystr뮠and J쩮, K.},
journal = {Information Processing \& Management},
number = {2},
pages = {191--213},
title = {{Task complexity affects information seeking and use}},
volume = {31},
year = {1995}
}
@inproceedings{Ahlberg1994,
author = {Ahlberg, C. and Shneiderman, B.},
title = {{Visual information seeking: Tight coupling of dynamic query filters with starfield displays}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {313--317},
year = {1994}
}
@article{Olson2003,
author = {Olson, C. and Chi, E. H.},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {3},
pages = {9--10},
title = {{ScentTrails: Integrating browsing and search on the web}},
volume = {10},
year = {2003}
}
@article{Kandel2011,
author = {Kandel, S. and Heer, J. and Plaisant, C. and Kennedy, J. and van Ham, F. and Riche, N. H. and Weaver, C. and Lee, B. and Brodbeck, D. and Buono, P.},
doi = {10.1177/1473871611415994},
journal = {Information Visualization},
number = {4},
pages = {271--288},
title = {{Research directions in data wrangling: Visualizations and transformations for usable and credible data}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415994},
volume = {10},
year = {2011}
}
@inproceedings{Abram1995,
author = {Abram, G. and Treinish, L.},
title = {{An extended data-flow architecture for data analysis and visualization}},
booktitle = {Proc. IEEE Conf. Visualization},
doi = {10.1109/VISUAL.1995.480821},
pages = {263--270},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=480821},
year = {1995}
}
@inproceedings{Endert2012a,
author = {Endert, A. and Fox, S. and Maiti, D. and Leman, S. and North, C.},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
title = {{The Semantics of Clustering: Analysis of User-Generated Spatializations of Text Documents}},
year = {2012}
}
@article{Shrinivasan2009,
author = {Shrinivasan, Y. B. and van Wijk, J. J.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {5},
pages = {34--43},
title = {{Supporting exploration awareness in information visualization}},
volume = {29},
year = {2009}
}
@article{Liu2008,
author = {Liu, Z. and Nersessian, N. and Stasko, J. T.},
doi = {10.1109/TVCG.2008.121},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1173--80},
title = {{Distributed cognition as a theoretical framework for information visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988961},
volume = {14},
year = {2008}
}
@inproceedings{Lee2005,
author = {Lee, B. and Czerwinski, M. and Robertson, G. and Bederson, B. B.},
title = {{Understanding research trends in conferences using paperLens}},
booktitle = {Proc. Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI '05)},
doi = {10.1145/1056808.1057069},
publisher = {ACM Press},
url = {http://portal.acm.org/citation.cfm?doid=1056808.1057069},
year = {2005}
}
@inproceedings{Hearst2006,
author = {Hearst, M.},
title = {{Design Recommendations for Hierarchical Faceted Search Interfaces}},
booktitle = {Proc. ACM SIGIR Workshop on Faceted Search},
year = {2006}
}
@article{Wattenberg2008,
author = {Wattenberg, M. and Vi\'{e}gas, F. B.},
doi = {10.1109/TVCG.2008.172},
journal = {IEEE trans. visualization and computer graphics (TVCG)},
number = {6},
pages = {1221--1228},
title = {{The Word Tree, an interactive visual concordance.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988967},
volume = {14},
year = {2008}
}
@inproceedings{Hearst1996,
author = {Hearst, M. and Pederson, J. O.},
title = {{Reexamining the Cluster Hypothesis: Scatter / Gather on Retrieval Results}},
booktitle = {Proc. ACM SIGIR},
pages = {76--84},
year = {1996}
}
@book{Bailey1994,
author = {Bailey, K. D.},
publisher = {SAGE Publications, Inc},
title = {{Typologies and Taxonomies: An Introduction to Classification}},
year = {1994}
}
@article{Monroe2008,
author = {Monroe, B. L. and Colaresi, M. P. and Quinn, K. M.},
doi = {10.1093/pan/mpn018},
issn = {1047-1987},
journal = {Political Analysis},
number = {4},
pages = {372--403},
title = {{Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict}},
url = {http://pan.oxfordjournals.org/cgi/doi/10.1093/pan/mpn018},
volume = {16},
year = {2008}
}
@article{Hopkins2010,
author = {Hopkins, D. J. and King, G.},
journal = {American Journal of Political Science},
number = {1},
pages = {229--247},
title = {{A Method of Automated Nonparametric Content Analysis for Social Science}},
volume = {54},
year = {2010}
}
@inproceedings{Hong2008,
author = {Hong, L. and Chi, E. H. and Budiu, R. and Pirolli, P. and Nelson, L.},
title = {{SparTag.us: A Low Cost Tagging System for Foraging of Web Content}},
booktitle = {Proc. ACM Advanced Visual Interfaces (AVI)},
pages = {65--72},
year = {2008}
}
@inproceedings{Bauer2006,
author = {Bauer, A. and Koedinger, K.},
title = {{Pasting and Encoding: Note-Taking in Online Courses}},
booktitle = {Proc. IEEE Intl. Conf. Advanced Learning Technologies (ICALT)},
doi = {10.1109/ICALT.2006.1652559},
pages = {789--793},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1652559},
year = {2006}
}
@incollection{Anderson1979,
title = {{An elaborative processing explanation of depth of processing}},
author = {Anderson, J. R. and Reder, L. M.},
booktitle = {Levlels of processing in human memory},
editor = {Cermak, L. S. and Craik, F. I. M.},
publisher = {Erlbaum},
year = {1979}
}
@article{Bradshaw1982,
author = {Bradshaw, G. L. and Anderson, J. R.},
doi = {10.1016/S0022-5371(82)90531-X},
journal = {Verbal Learning and Verbal Behavior},
number = {2},
pages = {165--174},
title = {{Elaborative encoding as an explanation of levels of processing}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S002253718290531X},
volume = {21},
year = {1982}
}
@article{Bretzinc1979,
author = {Bretzinc, B. H. and Kulhavy, R.},
journal = {Contemporary Educational Psychology},
pages = {145--153},
title = {{Note鮧 and Depth of Processing}},
volume = {4},
year = {1979}
}
@inproceedings{Bodnar2005,
author = {Bodnar, J.},
title = {{Making Sense of Massive Data by Hypothesis Testing}},
booktitle = {Proc. Intl. Conf. Intelligence Analysis},
pages = {2--4},
year = {2005}
}
@article{Green2009,
author = {Green, T. M. and Ribarsky, W. and Fisher, B.},
doi = {10.1057/ivs.2008.28},
journal = {Information Visualization},
number = {1},
pages = {1--13},
title = {{Building and applying a human cognition model for visual analytics}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2008.28},
volume = {8},
year = {2009}
}
@inproceedings{Green2008,
author = {Green, T. M. and Ribarsky, W. and Fisher, B.},
title = {{Visual analytics for complex concepts using a human cognition model}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2008.4677361},
pages = {91--98},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4677361},
year = {2008}
}
@article{Heer2008a,
author = {Heer, J. and Agrawala, M.},
doi = {10.1057/palgrave.ivs.9500167},
journal = {Information Visualization},
number = {1},
pages = {49--62},
title = {{Design considerations for collaborative visual analytics}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500167},
volume = {7},
year = {2008}
}
@inproceedings{Collins2009,
author = {Collins, C. and Carpendale, S. and Penn, G.},
title = {{DocuBurst: Visualizing Document Content using Language Structure}},
booktitle = {Proc. Eurographics / IEEE-VGTC Symp. Visualization (EuroVIs)},
number = {3},
pages = {1039--1046},
volume = {28},
year = {2009}
}
@inproceedings{Lee2012,
author = {Lee, H. and Lee, S. and Kim, N. and Seo, J.},
title = {{JigsawMap: Connecting the Past to the Future by Mapping Historical Textual Cadasters}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {463--472},
year = {2012}
}
@inproceedings{Muralidharan2012,
author = {Muralidharan, A. and Hearst, M.},
title = {{A Sensemaking Environment for Literature Study}},
booktitle = {Extended Abstracts of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1955--1960},
year = {2012}
}
@article{Ramsay2010,
author = {Ramsay, S.},
title = {{The Hermeneutics of Screwing Around; or What You Do with a Million Books}},
url = {http://www.playingwithhistory.com/wp-content/uploads/2010/04/hermeneutics.pdf},
year = {2010}
}
@book{Beveridge1958,
author = {Beveridge, W. I. B.},
edition = {Revised ed},
publisher = {W. W. Norton and Company},
title = {{The art of scientific investigation.}},
url = {http://www.cabdirect.org/abstracts/19522203234.html},
year = {1958}
}
@book{Keim2010,
author = {Keim, D. A. and Kohlhammer, J. and Ellis, G. and Mansmann, F.},
publisher = {Eurographics Association},
title = {{Mastering The Information Age-Solving Problems with Visual Analytics}},
url = {http://books.google.com/books?hl=en\&lr=\&id=vdv5wZM8ioIC\&oi=fnd\&pg=PA1\&dq=Mastering+the+Information+Age:+Solving+Problems+with+Visual+Analytics\&ots=yQ2iF59lCM\&sig=kVHMhbfnBGwq7Vg6ztl07vmlI-g},
year = {2010}
}
@inproceedings{Chuang2012,
author = {Chuang, J. and Ramage, D. and Manning, C. D. and Heer, J.},
title = {{Interpretation and Trust: Designing Model-Driven Visualizations for Text Analysis}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
year = {2012}
}
@inproceedings{Wise1995,
author = {Wise, JA and Thomas, JJ and Pennock, K.},
title = {{Visualizing the non-visual: Spatial analysis and interaction with information from text documents}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=528686},
year = {1995}
}
@inproceedings{Endert2011,
author = {Endert, A. and Han, C. and Maiti, D. and House, L. and Leman, S. C. and North, C.},
title = {{Observation-level interaction with statistical models for visual analytics}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology (VAST)},
pages = {121--130},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=6102449},
year = {2011}
}
@inproceedings{Gotz2006a,
author = {Gotz, D. and Zhou, M. and Aggarwal, V.},
title = {{Interactive Visual Synthesis of Analytic Knowledge}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2006.261430},
pages = {51--58},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4035747},
year = {2006}
}
@inproceedings{Callahan2006,
author = {Callahan, S. P. and Freire, J. and Santos, E.},
title = {{VisTrails: visualization meets data management}},
booktitle = {Proc. ACM Intl. Conf. Management of Data (SIGMOD)},
pages = {745--747},
url = {http://dl.acm.org/citation.cfm?id=1142574},
year = {2006}
}
@article{Jeong2009,
author = {Jeong, D. H. and Ziemkiewicz, C. and Fisher, B. and Ribarsky, W. and Chang, R.},
journal = {Computer Graphics Forum},
number = {3},
pages = {767--774},
title = {{iPCA: An Interactive System for PCA-based Visual Analytics}},
volume = {28},
year = {2009}
}
@inproceedings{Fisher2012,
author = {Fisher, K. and Counts, S. and Kittur, A. and Ave, F.},
title = {{Distributed Sensemaking: Improving Sensemaking by Leveraging the Efforts of Previous Users}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
year = {2012}
}
@inproceedings{Robinson2008,
author = {Robinson, A. C.},
title = {{Collaborative Synthesis of Visual Analytic Results}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology (VAST '08)},
isbn = {9781424429356},
pages = {67--74},
year = {2008}
}
@article{Lehrer2008,
author = {Lehrer, J.},
title = {{The Eureka Hunt: Why do good ideas come to us when they do?}},
journal = {The New Yorker},
pages = {40--45},
year = {2008}
}
@book{Heuer1999,
address = {Washington, DC, USA},
author = {Heuer, R.},
title = {{Psychology of Intelligence Analysis}},
publisher = {Central Intelligence Agency},
year = {1999}
}
@book{Johnson2005,
title = {{Analytic Culture in the US Intelligence Community: An Ethnographic Study}},
address = {Washington, DC, USA},
author = {Johnson, R.},
publisher = {Central Intelligence Agency},
year = {2005}
}
@article{Mai2004,
author = {Mai, X. Q. and Luo, J. and Wu, J. H. and Luo, Y. J.},
title = {{Aha! effects in a guessing riddle task: an event-related potential study.}},
journal = {Human brain mapping},
number = {4},
pages = {261--70},
volume = {22},
year = {2004}
}
@inproceedings{Gotz2006,
author = {Gotz, D. and Zhou, M. X. and Wen, Z.},
title = {{Interactive Visual Synthesis of Analytic Knowledge}},
booktitle = {Proc. IUI '06 Workshop on IUI for Intelligence Analysis},
year = {2006}
}
@inproceedings{Chin2009,
author = {Chin, G. and Kuchar, O. A. and Wolf, K. E.},
title = {{Exploring the Analytical Processes of Intelligence Analysts}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '09)},
pages = {11--20},
year = {2009}
}
@article{Alter2007,
author = {Alter, A. L. and Oppenheimer, D. M. and Epley, N. and Eyre, R. N.},
doi = {10.1037/0096-3445.136.4.569},
journal = {Journal of experimental psychology, General},
number = {4},
pages = {569--576},
title = {{Overcoming intuition: metacognitive difficulty activates analytic reasoning.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17999571},
volume = {136},
year = {2007}
}
@article{Eick2000,
author = {Eick, S. G.},
doi = {10.1109/2945.841120},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {1},
pages = {44-58},
title = {{Visual discovery and analysis}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=841120},
volume = {6},
year = {2000}
}
@article{Russell1993,
address = {New York, New York, USA},
title = {{The cost structure of sensemaking}},
author = {Russell, D. M. and Stefik, M. J. and Pirolli, P. and Card, S. K.},
doi = {10.1145/169059.169209},
journal = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '93)},
pages = {269--276},
publisher = {ACM Press},
url = {http://portal.acm.org/citation.cfm?doid=169059.169209},
year = {1993}
}
@inproceedings{Byrne1999,
author = {Byrne, M. D. and John, B. E. and Wehrle, N. S. and Crow, D. C.},
title = {{The Tangled Web We Wove : A Taskonomy of WWW Use}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '99)},
pages = {533--551},
year = {1999}
}
@article{Golovchinsky1999,
author = {Golovchinsky, G. and Belkin, N. J.},
title = {{Innovation and Evaluation of Information: A CHI '98 Workshop}},
journal = {SIGCHI Bulletin},
number = {1},
pages = {22--25},
volume = {31},
year = {1999}
}
@book{Hoffman2007,
title={Expertise out of context: proceedings of the sixth International Conference on Naturalistic Decision Making},
author={Hoffman, R.R.},
series={Expertise: Research and Applications Series},
url={http://books.google.ca/books?id=04bWo3CTamQC},
year={2007},
publisher={Lawrence Erlbaum Associates}
}
@article{Pirolli1999,
author = {Pirolli, P. and Card, S.},
doi = {10.1037/0033-295X.106.4.643},
title = {{Information Foraging}},
journal = {Psychological Review},
number = {4},
pages = {643--675},
publisher = {American Psychological Association},
url = {http://doi.apa.org/getdoi.cfm?doi=10.1037/0033-295X.106.4.643},
volume = {106},
year = {1999}
}
@article{Klein2006a,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 1: Alternative Perspectives}},
journal = {IEEE Intelligent Systems},
number = {4},
pages = {70--73},
volume = {21},
year = {2006}
}
@article{Klein2006,
author = {Klein, G. and Moon, B. and Hoffman, R. R.},
title = {{Making Sense of Sensemaking 2: A Macrocognitive Model}},
journal = {IEEE Intelligent Systems},
number = {5},
pages = {88--92},
volume = {21},
year = {2006}
}
@inproceedings{Scholtz2008,
author = {Scholtz, J.},
booktitle = {Proc. ACM Computer Human Information (CHI) conference Workshop on Sensemaking},
title = {{Progress and Challenges in Evaluating Tools for Sensemaking}},
year = {2008}
}
@book{Manning2009,
title = {{An Introduction to Information Retrieval}},
author = {Manning, C. D.. and Raghavan, P. and Schutze, H.},
pages = {1--544},
publisher = {Cambridge University Press},
year = {2009}
}
}}}
{{{
@article{Ziemkiewicz2010,
author = {Ziemkiewicz, C. and Kosara, R.},
title = {{Implied dynamics in information visualization}},
journal = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {215--222},
year = {2010}
}
@article{BrownE2014,
author = {{Brown, E}, T. and Ottley, A. and Zhao, H. and Lin, Q. and Souvenir, R. and Endert, A. and Chang, R.},
journal = {To appear in IEEE TVCG (Proc. VAST)},
title = {{Finding Waldo: Learning about users from their interactions}},
year = {2014}
}
@article{Talbot2014,
author = {Talbot, J. and Setlur, V. and Anand, A.},
journal = {To appear in IEEE TVCG (Proc. InfoVis)},
title = {{Four experiments on the perception of bar charts}},
year = {2014}
}
@techreport{Isenberg2014,
author = {Isenberg, P. and Isenberg, T. and Sedlmair, M. and Chen, J. and M欬er, T.},
title = {{Toward a deeper understanding of visualization through keyword analysis}},
institution = {Inria},
pages = {1--24},
year = {2014}
}
@article{Correll2014,
author = {Correll, M. and Gleicher, M.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Error bars considered harmful: Exploring alternate encodings for mean and error}},
year = {2014}
}
@article{Armstrong2014,
author = {Armstrong, Z. and Wattenberg, M.},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
title = {{Visualizing statistical mix effects and Simpson鳠paradox}},
year = {2014}
}
@article{Sedlmair2014,
author = {Sedlmair, M. and Heinzl, C. and Bruckner, S. and Piringer, H. and M\"{o}ller, T.},
title = {{Visual parameter space analysis: A conceptual framework}},
journal = {To Appear in IEEE TVCG (Proc. InfoVis)},
year = {2014}
}
@article{Kuang2012,
author = {Kuang, X. and Zhang, H. and Zhao, S. and McGuffin, M.J.},
title = {{Tracing Tuples Across Dimensions: A Comparison of Scatterplots and Parallel Coordinate Plots}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3pt4},
pages = {1365--1374},
volume = {31},
year = {2012}
}
@article{Isenberg2013b,
author = {Isenberg, T. and Isenberg, P. and Chen, J. and Sedlmair, M. and M쥲, T.},
title = {{A systematic review on the practice of evaluating visualization}},
doi = {10.1109/TVCG.2013.126},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2818--2827},
url = {http://www.ncbi.nlm.nih.gov/pubmed/24051849},
volume = {19},
year = {2013}
}
@inproceedings{Thudt2013,
author = {Thudt, A. and Baur, D. and Carpendale, S.},
title = {{Visits: A spatiotemporal visualization of location histories}},
booktitle = {Proc. Eurographics Conf. Visualization (EuroVis)},
year = {2013}
}
@inproceedings{Sedlmair2011a,
author = {Sedlmair, M. and Isenberg, P. and Baur, D. and Mauerer, M. and Pigorsch, C. and Butz, A.},
title = {{Cardiogram: Visual analytics for automotive engineers}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
keywords = {`},
pages = {1727--1736},
year = {2011}
}
@inproceedings{Neumann2007,
author = {Neumann, P. and Tat, A and Zuk, T. and Carpendale, S.},
title = {{KeyStrokes: Personalizing typed text with visualization}},
booktitle = {Proc. Joint Eurographics (EuroVis)/ IEEE VGTC Conf. Visualization},
pages = {43--50},
year = {2007}
}
@article{Shrinivasan2010,
author = {Shrinivasan, Y. B. and {Van Wijk}, J. J.},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {803--812},
title = {{Supporting exploratory analysis with the select \& slice table}},
volume = {29},
year = {2010}
}
@article{Pretorius2008,
author = {Pretorius, A. J. and van Wijk, J. J.},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
pages = {967--974},
title = {{Visual inspection of multivariate graphs}},
volume = {27},
year = {2008}
}
@article{Erbacher2010,
author = {Erbacher, R.F. and Frincke, D. A. and {Chung Wong}, P. and Moody, S. and Fink, G.},
journal = {Information Visualization},
number = {3},
pages = {204--219},
title = {{A multi-phase network situational awareness cognitive task analysis}},
volume = {9},
year = {2010}
}
@article{Xie2009b,
author = {Xie, S. and Kraemer, E. and Stirewalt, R. E. K. and Dillon, L. K. and Fleming, S. D.},
journal = {Information Visualization},
number = {2},
pages = {120--136},
title = {{Design and evaluation of extensions to UML sequence diagrams for modeling multithreaded interactions}},
volume = {8},
year = {2009}
}
@article{Fatto2008,
author = {Fatto, D. D. and Laurini, R. and Lopez, K. and Sebillo, M. and Vitiello, G.},
journal = {Information Visualization},
number = {3-4},
pages = {253--264},
title = {{A chorem-based approach for visually synthesizing complex phenomena}},
volume = {7},
year = {2008}
}
@inproceedings{Brewer2000,
author = {Brewer, I. and MacEachren, A.M. and Abdo, H. and Gundrum, J. and Otto, G.},
title = {{Collaborative geographic visualization: Enabling shared understanding of environmental processes}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {137--141},
volume = {2000},
year = {2000}
}
@article{Lundstrom2011,
author = {Lundstr젃. and Rydell, T. and Forsell, C. and Persson, A. and Ynnerman, A.},
title = {{Multi-touch table system for medical visualization: application to orthopedic surgery planning.}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {1775--84},
volume = {17},
year = {2011}
}
@article{Gotz2009,
author = {Gotz, D. and Zhou, M. X.},
title = {{Characterizing users橳ual analytic activity for insight provenance}},
journal = {Information Visualization},
number = {1},
pages = {42--55},
volume = {8},
year = {2009}
}
@article{Wang2007,
author = {Wang, W. and Lu, A.},
title = {{Interactive wormhole detection and evaluation}},
journal = {Information Visualization},
number = {1},
pages = {3--17},
volume = {6},
year = {2007}
}
@article{Kincaid2005,
author = {Kincaid, R. and Ben-Dor, A. and Yakhini, Z.},
title = {{Exploratory visualization of array-based comparative genomic hybridization}},
journal = {Information Visualization},
number = {3},
pages = {176--190},
volume = {4},
year = {2005}
}
@article{McKeon2009,
author = {McKeon, M.},
title = {{Harnessing the web information ecosystem with wiki-based visualization dashboards}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1081--1088},
volume = {15},
year = {2009}
}
@inproceedings{Robinson2008,
author = {Robinson, A. C.},
title = {{Collaborative synthesis of visual analytic results}},
booktitle = {Proc. IEEE Symp Visual Analytics Science and Technology (VAST)},
pages = {67--74},
year = {2008}
}
@inproceedings{Laramee2013,
author = {Laramee, R. S. and Chen, M. and Ebert, D. and Fisher, B. and Munzner, T.},
title = {{Evaluation: How much evaluation is enough?}},
booktitle = {IEEE VIS Panel},
year = {2013}
}
@article{Isenberg2013,
author = {Isenberg, T. and Isenberg, P. and Chen, J. and Sedlmair, M. and M쥲, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{A systematic review on the practice of evaluating visualization}},
volume = {To Appear},
year = {2013}
}
@article{VanWijk2013,
author = {van Wijk, J. J.},
doi = {10.1109/MC.2013.151},
journal = {IEEE Computer},
pages = {1--8},
title = {{Evaluation: A challenge for visual analytics}},
url = {http://doi.ieeecomputersociety.org/10.1109/MC.2013.151},
volume = {In press},
year = {2013}
}
@article{Scholtz2013,
author = {Scholtz, J. and Plaisant, C. and Whiting, M. and Grinstein, G.},
doi = {10.1177/1473871613490290},
journal = {Information Visualization},
title = {{Evaluation of visual analytics environments: The road to the Visual Analytics Science and Technology challenge evaluation methodology}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871613490290}
year = {(in press)}
}
@inproceedings{Erickson2012,
title = {{The Dubuque water portal: Evaluation of the uptake, use and impact of residential water consumption feedback}},
author = {Erickson, T. and Podlaseck, M. and Sahu, S. and Dai, J. D. and Chao, T. and Naphade, M.},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2207772},
pages = {675--684},
year = {2012}
}
@article{Boukhelifa2013,
author = {Boukhelifa, N. and Cancino, W. and Bezerianos, A. and Lutton, E.},
title = {{Evolutionary visual exploration: Evaluation with expert users}},
journal = {Computer Graphics Forum (Proc. EuroVis)},
number = {3},
volume = {32},
year = {2013}
}
@article{Perer2012,
author = {Perer, A. and Guy, I. and Uziel, E. and Ronen, I. and Jacovi, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
title = {{The longitudinal use of SaNDVis: Visual social network analytics in the enterprise.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/23248134},
volume = {to appear},
year = {2013}
}
@article{Fang2012,
author = {Fang, H. and Tam, G. K. and Borgo, R. and Aubrey, A. and Grant, P. and Rosin, P. and Wallraven, C. and Cunningham, D. and Marshall, A. D. and Chen, M.},
doi = {10.1109/TVCG.2012.312},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {99},
title = {{Visualizing natural image statistics}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6361387},
volume = {PP},
year = {2012}
}
@techreport{Wickham2012,
author = {Wickham, H.},
pages = {1--8},
title = {{Graphical criticism: Some historical notes}},
url = {http://vita.had.co.nz/papers/stat-graph-hist.pdf},
year = {2012}
}
@inproceedings{Lipford2010,
author = {Lipford, H. R. and Stukes, F. and Dou, W. and Hawkins, M. E. and Chang, R.},
title = {{Helping users recall their reasoning process}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2010.5653598},
pages = {187--194},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5653598},
year = {2010}
}
@article{Isenberg2012,
author = {Isenberg, P. and Fisher, D. and Paul, S. A. and Morris, M. R. and Inkpen, K. and Czerwinski, M.},
doi = {10.1109/TVCG.2011.287},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {689--702},
title = {{Co-located collaborative visual analytics around a tabletop display.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22156107},
volume = {18},
year = {2012}
}
@inproceedings{Scholtz2006,
author = {Scholtz, J.},
title = {{Beyond usability: Evaluation aspects of visual analytic environments}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
doi = {10.1109/VAST.2006.261416},
pages = {145--150},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4035759},
year = {2006}
}
@inproceedings{Bandlow2011,
author = {Bandlow, A. and Matzen, L. E. and Cole, K. S. and Dornburg, C. C. and Geiseler, C. J. and Greenfield, J. A. and McNamara, L. and Stevens-Adams, S. M.},
title = {{Evaluating information visualizations with working memory metrics}},
booktitle = {Posters Extended Abstracts of HCI Intl.},
doi = {10.1007/978-3-642-22098-2\_53},
pages = {265--269},
year = {2011}
}
@article{Haroz2012,
author = {Haroz, S. and Whitney, D.},
doi = {10.1109/TVCG.2012.233},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2402--2410},
title = {{How capacity limits of attention influence information visualization effectiveness}},
url = {http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=\&arnumber=6327245\&contentType=Journals+\%26+Magazines\&queryText\%3DHow+capacity+limits+of+attention+influence+information+visualization+effectiveness},
volume = {18},
year = {2012}
}
@article{Gomez2012a,
author = {Gomez, S. R. and Jianu, R. and Ziemkiewicz, C. and Guo, H. and Laidlaw, D. H.},
doi = {http://dx.doi.org/10.1109/TVCG.2012.214},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2411--2420},
title = {{Different strokes for different folks: visual presentation design between disciplines}},
url = {http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=\&arnumber=6327246\&contentType=Journals+\%26+Magazines\&queryText\%3DDifferent+Strokes+for+Different+Folks+\%3A+Visual+Presentation+Design+between+Disciplines},
volume = {18},
year = {2012}
}
@article{Hofmann2012,
author = {Hofmann, H. and Follett, L. and Majumder, M. and Cook, D.},
doi = {http://dx.doi.org/10.1109/TVCG.2012.230},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2441-- 2448},
title = {{Graphical tests for power comparison of competing designs}},
url = {http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=\&arnumber=6327249\&contentType=Journals+\%26+Magazines\&queryText\%3DGraphical+Tests+for+Power+Comparison+of+Competing+Designs},
volume = {18},
year = {2012}
}
@inproceedings{Arias-Hernandez2011,
author = {Arias-Hernandez, R. and Kaastra, L. T. and Green, T. M. and Fisher, B.},
doi = {10.1109/HICSS.2011.339},
month = jan,
pages = {1--10},
title = {{Pair analytics: Capturing reasoning processes in collaborative visual analytics}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5718616},
booktitle = {Proc. Hawaii Intl. Conf System Sciences (HICSS)},
year = {2011}
}
@techreport{Merkelbach1994,
author = {Merkelbach, E. J. H. M. and Schraagen, J. M. C.},
institution = {TNO Human Factors Research Institute: Information Processing},
title = {{A framework for the analysis of cognitive tasks}},
url = {http://handle.dtic.mil/100.2/ADA285345},
year = {1994}
}
@article{Migut2012,
author = {Migut, M. and Worring, M.},
doi = {10.1177/1473871611433715},
journal = {Information Visualization},
number = {3},
pages = {237--251},
title = {{Visual exploration of classification models for various data types in risk assessment}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611433715},
volume = {11},
year = {2012}
}
@article{Hao2012,
author = {Hao, M. C. and Marwah, M. and Janetzko, H. and Dayal, U. and Keim, D. A. and Patnaik, D. and Ramakrishnan, N. and Sharma, R. K.},
doi = {10.1177/1473871611430769},
journal = {Information Visualization},
number = {1},
pages = {71--83},
title = {{Visual exploration of frequent patterns in multivariate time series}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611430769},
volume = {11},
year = {2012}
}
@article{Green2012,
author = {Green, T. M. and Fisher, B.},
doi = {10.1177/1473871612441542},
journal = {Information Visualization},
number = {3},
pages = {205--221},
title = {{Impact of personality factors on interface interaction and the development of user profiles: Next steps in the personal equation of interaction}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871612441542},
volume = {11},
year = {2012}
}
@article{Dou2012,
author = {Dou, W. and Ziemkiewicz, C. and Harrison, L. and Jeong, D. H. and Ribarsky, W. and Wang, X. and Chang, R.},
doi = {10.1177/1473871611433712},
journal = {Information Visualization},
number = {3},
pages = {222--236},
title = {{Toward a deeper understanding of the relationship between interaction constraints and visual isomorphs}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611433712},
volume = {11},
year = {2012}
}
@article{Crampton2002,
author = {Crampton, J. W.},
doi = {10.1559/152304002782053314},
journal = {Cartography and Geographic Information Science},
number = {2},
pages = {85--98},
title = {{Interactivity types in geographic visualization}},
url = {http://openurl.ingenta.com/content/xref?genre=article\&issn=1523-0406\&volume=29\&issue=2\&spage=85},
volume = {29},
year = {2002}
}
@incollection{Edsall1995,
author = {Edsall, R. and Andrienko, G. and Andrienko, N. and Buttenfield, B.},
title = {{Interactive maps for exploring spatial data}},
booktitle = {Manual of Geographic Information Systems},
chapter = {42},
editor = {Madden, M.},
pages = {837--858},
publisher = {ASPRS},
year = {1995}
}
@article{MacEachren1999,
author = {MacEachren, A. M. and Wachovitz, M. and Edsall, R. and Haug, D. and Masters, R.},
journal = {Intl. J. Geographical Information Science},
number = {4},
pages = {311--334},
title = {{Constructing knowledge from multivariate spatiotemporal data: Integrating geographical visualization with knowledge discovery in database methods}},
volume = {13},
year = {1999}
}
@article{Andrienko2003,
author = {Andrienko, N. and Andrienko, G. and Gatalsky, P.},
doi = {10.1016/S1045-926X(03)00046-6},
journal = {Visual Languages \& Computing},
number = {6},
pages = {503--541},
title = {{Exploratory spatio-temporal visualization: An analytical review}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1045926X03000466},
volume = {14},
year = {2003}
}
@inproceedings{Bautista2008,
author = {Bautista, J. and Carenini, G.},
title = {{An empirical evaluation of interactive visualizations for preferential choice}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/1385569.1385603},
pages = {207--214},
url = {http://portal.acm.org/citation.cfm?doid=1385569.1385603},
year = {2008}
}
@inproceedings{Bautista2006,
author = {Bautista, J. and Carenini, G.},
title = {{An integrated task-based framework for the design and evaluation of visualizations to support preferential choice}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
doi = {10.1145/1133265.1133308},
pages = {217--224},
url = {http://portal.acm.org/citation.cfm?doid=1133265.1133308},
year = {2006}
}
@article{Zhang1996,
author = {Zhang, J.},
journal = {Intl. J. Human-Computer Studies},
pages = {59--74},
title = {{A representational analysis of relational information displays}},
volume = {45},
year = {1996}
}
@inproceedings{Espinosa1999,
author = {Espinosa, O. J. and Hendrickson, C. and Garrett, J. H.},
title = {{Domain analysis: A technique to design a user-centered visualization framework}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {44--52},
year = {1999}
}
@inproceedings{Treinish1999,
author = {Treinish, L. A.},
title = {{A function-based data model for visualization}},
booktitle = {Proc. IEEE Conf. Visualization Late Breaking Hot Topics},
pages = {73--76},
year = {1999}
}
@inproceedings{Kreuseler2004,
author = {Kreuseler, M. and Nocke, T. and Schumann, H.},
title = {{A history mechanism for visual data mining}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
doi = {10.1109/INFVIS.2004.2},
pages = {49--56},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1382890},
year = {2004}
}
@inproceedings{Jankun-Kelly2002,
author = {Jankun-Kelly, T. J. and Ma, K. L. and Gertz, M.},
title = {{A model for the visualization exploration process}},
booktitle = {Proc. IEEE Conf. Visualization},
doi = {10.1109/VISUAL.2002.1183791},
pages = {323--330},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1183791},
year = {2002}
}
@article{Rheingans2002,
author = {Rheingans, P. L.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {1},
pages = {6--10},
title = {{Are we there yet? Exploring with dynamic visualization}},
volume = {22},
year = {2002}
}
@article{Upson1989,
author = {Upson, C. and Faulhaber, T. and Laidlaw, D. and Schlegel, D. and Gurwitz, R. and van Dam, A.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {4},
pages = {30--42},
title = {{The application visualization system: A computational environment for scientific visualization}},
volume = {9},
year = {1989}
}
@article{Duke2005,
author = {Duke, D. J. and Brodlie, K. W. and Duce, D. A. and Herman, I.},
journal = {IEEE Trans. Computer Graphics and Applications (TCGA)},
number = {3},
pages = {6--9},
title = {{Do you see what I mean?}},
volume = {25},
year = {2005}
}
@article{Jarvenpaa1988,
author = {Jarvenpaa, S. L. and Dickson, G. W.},
journal = {Communications of the ACM},
number = {6},
pages = {764--774},
title = {{Graphics and managerial decision making: Research based guidelines}},
volume = {37},
year = {1988}
}
@inproceedings{Casner1989,
author = {Casner, S. M. and Larkin, J. H.},
title = {{Cognitive efficiency considerations for good graphic design}},
booktitle = {Proc. Conf. Cognitive Science Soc.},
year = {1989}
}
@article{Larkin1987,
author = {Larkin, J. H. and Simon, H. A.},
journal = {Cognitive Science},
pages = {65--99},
title = {{Why a diagram is (sometimes) worth ten thousand words}},
volume = {11},
year = {1987}
}
@article{Kieras1985,
author = {Kieras, D. and Polson, P. G.},
journal = {Man-Machine Studies},
pages = {365--394},
title = {{An approach to the formal analysis of user complexity}},
volume = {22},
year = {1985}
}
@article{Koedinger1990,
author = {Koedinger, K. R. and Anderson, J. R.},
doi = {10.1207/s15516709cog1404\_2},
journal = {Cognitive Science},
number = {4},
pages = {511--550},
title = {{Abstract planning and perceptual chunks: Elements of expertise in geometry}},
url = {http://doi.wiley.com/10.1207/s15516709cog1404\_2},
volume = {14},
year = {1990}
}
@article{Hornbඬ
author = {Hornb뮽,
doi = {10.1016/j.ijhcs.2005.06.002},
journal = {Intl. J. Human-Computer Studies},
number = {2},
pages = {79--102},
title = {{Current practice in measuring usability: Challenges to usability studies and research}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581905001138},
volume = {64},
year = {2006}
}
@article{Kennedy1996,
author = {Kennedy, J. B. and Mitchell, K. J. and Barclay, P. J.},
journal = {ACM SIGMOD Record},
pages = {30--34},
title = {{A framework for information visualisation}},
volume = {25},
year = {1996}
}
@inproceedings{Rogowitz1993,
author = {Rogowitz, B. E. and Treinish, L. A.},
title = {{An architecture for rule-based visualization}},
booktitle = {Proc. IEEE Conf. Visualization},
doi = {10.1109/VISUAL.1993.398874},
pages = {236--243},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=398874},
year = {1993}
}
@inproceedings{Lohse1990,
author = {Lohse, J. and Rueter, H. and Biolsi, K. and Walker, N.},
title = {{Classifying visual knowledge representations: A foundation for visualization research}},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {131--138},
year = {1990}
}
@inproceedings{Lee1995,
author = {Lee, J. P. and Grinstein, G. G.},
title = {{An architecture for retaining and analyzing visual explorations of databases}},
booktitle = {Proc. IEEE Visualization},
pages = {101--108},
year = {1995}
}
@inproceedings{Card1997,
author = {Card, S. K. and Mackinlay, J. D.},
title = {{The structure of the information visualization design space}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
pages = {92--99},
year = {1997}
}
@article{Lohse1994,
author = {Lohse, G. L. and Biolsi, K. and Walker, N. and Rueter, H. H.},
doi = {10.1145/198366.198376},
journal = {Communications of the ACM},
number = {12},
pages = {36--49},
title = {{A classification of visual representations}},
url = {http://portal.acm.org/citation.cfm?doid=198366.198376},
volume = {37},
year = {1994}
}
@article{Kosslyn1989,
author = {Kosslyn, S. M.},
journal = {Applied Cognitive Psychology},
pages = {185--226},
title = {{Understanding charts and graphs}},
volume = {2},
year = {1989}
}
@inproceedings{Roth1991,
author = {Roth, S. F. and Mattis, J.},
title = {{Automating the presentation of information}},
booktitle = {Proc. IEEE Conf. Artificial Intelligence Applications},
doi = {10.1109/CAIA.1991.120851},
pages = {90--97},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=120851},
year = {1991}
}
@article{Ignatius1994,
author = {Ignatius, E. and Senay, H. and Favre, J.},
journal = {Visual Language and Computing},
pages = {321--338},
title = {{An intelligent system for task-specific visualization assistance}},
volume = {5},
year = {1994}
}
@inproceedings{Ward2004,
author = {Ward, M. and Yang, J.},
title = {{Interaction spaces in data and information visualization}},
booktitle = {Proc. IEEE Symp. Information Visualization (InfoVis)},
number = {3},
pages = {137--145},
volume = {22},
year = {2004}
}
@inproceedings{Tweedie1997,
author = {Tweedie, L.},
title = {{Characterizing interactive externalizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/258549.258803},
pages = {375--382},
url = {http://portal.acm.org/citation.cfm?doid=258549.258803},
year = {1997}
}
@inproceedings{Dix1998,
author = {Dix, A. and Ellis, G.},
title = {{Starting simple: Adding value to static visualization through simple interaction}},
booktitle = {Proc. ACM Conf. Advanced Visual Interfaces (AVI)},
pages = {124--134},
year = {1998}
}
@article{Buja1996,
author = {Buja, A. and Cook, D. and Swayne, D. F.},
journal = {Computational and Graphical Statistics},
number = {1},
pages = {78--99},
title = {{Interactive high-dimensional data visualization}},
volume = {5},
year = {1996}
}
@article{Keim2002,
author = {Keim, D. A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
pages = {1--8},
title = {{Information visualization and visual data mining}},
volume = {8},
year = {2002}
}
@article{Conversy2011a,
author = {Conversy, S. and Chatty, S. and Hurter, C.},
doi = {10.1177/1473871611415988},
journal = {Information Visualization},
number = {3},
pages = {196--211},
title = {{Visual scanning as a reference framework for interactive representation design}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415988},
volume = {10},
year = {2011}
}
@inproceedings{Wang2011b,
author = {Wang, X. and Dou, W. and Butkiewicz, T and Bier, E. A. and Ribarsky, W.},
title = {{A two-stage framework for designing visual analytics system in organizational environments}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {249--258},
year = {2011}
}
@article{Hullman2011b,
author = {Hullman, J. and Diakopoulos, N.},
doi = {10.1109/TVCG.2011.255},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2231--2240},
title = {{Visualization rhetoric: Framing effects in narrative visualization}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034342},
volume = {17},
year = {2011}
}
@article{Spritzer2012,
author = {Spritzer, A. S. and Freitas, C. M. D. S.},
doi = {10.1109/TVCG.2011.106},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {822--835},
title = {{Design and evaluation of MagnetViz: A graph visualization tool}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/21690654},
volume = {18},
year = {2012}
}
@article{Ham2009,
author = {van Ham, F. and Perer, A.},
doi = {10.1109/TVCG.2009.108},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {953--960},
title = {{"Search, show context, expand on demand": supporting large graph exploration with degree-of-interest.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834159},
volume = {15},
year = {2009}
}
@article{Heer2008b,
author = {Heer, J. and Mackinlay, J. D. and Stolte, C. and Agrawala, M.},
doi = {10.1109/TVCG.2008.137},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {6},
pages = {1189--1196},
title = {{Graphical histories for visualization: supporting analysis, communication, and evaluation}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988963},
volume = {14},
year = {2008}
}
@inproceedings{Casner1991a,
author = {Casner, S. M.},
title = {{BOZ: Task-Analytic Design and Evaluation of Visual Displays}},
booktitle = {Proc. IEEE Intl. Conf. Systems, Man, and Cybernetics (SMC)},
pages = {1195--1200},
volume = {00},
year = {1991}
}
@article{Wills2008,
author = {Wills, G. and Wilkinson, L.},
doi = {10.1057/ivs.2008.27},
journal = {Information Visualization},
number = {1},
pages = {47--69},
title = {{AutoVis: Automatic visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/ivs.2008.27},
volume = {9},
year = {2008}
}
@article{Sorokin2008,
author = {Sorokin, A. and Forsyth, D. and Goodwin, N.},
journal = {Computer Vision and Pattern Recognition Workshops},
pages = {1--8},
title = {{Utility data annotation with Amazon Mechanical Turk}},
year = {2008}
}
@inproceedings{Viegas2008,
author = {Vi젆. B. and Wattenberg, M. and Mckeon, M. and Ham, F. V. and Kriss, J.},
title = {{Harry Potter and the Meat-Filled Freezer: A Case Study of Spontaneous Usage of Visualization Tools}},
booktitle = {Proc. HICSS},
publisher = {Citeseer},
year = {2008}
}
@inproceedings{Oleson2011,
author = {Oleson, D. and Sorokin, A. and Laughlin, R. and Hester, V. and Le, J. and Biewald, L.},
title = {{Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing}},
booktitle = {Proc. HComp},
pages = {43--48},
year = {2011}
}
@article{Kosara2003,
author = {Kosara, R. and Healey, C. G. and Interrante, V. and Laidlaw, D. H. and Ware, C.},
journal = {IEEE Computer Graphics and Applications},
number = {4},
pages = {20--25},
title = {{Thoughts on user studies: Why, how, and when}},
url = {http://visplore.net/publications/pdfs/VRVis\_2005\_11\_03\_14\_12\_04.pdf},
volume = {23},
year = {2003}
}
@inproceedings{Rester2005,
author = {Rester, M. and Pohl, M. and Hinum, K. and Miksch, S. and Ohmann, S. and Banovic, S.},
title = {{Assessing the Usability of an Interactive Information Visualization Method as the First Step of a Sustainable Evaluation}},
booktitle = {Proc. Empowering Software Quality.},
pages = {31--44},
publisher = {Austrian Computer Society},
year = {2005}
}
@inproceedings{Ziemkiewicz2009,
author = {Ziemkiewicz, C. and Kosara, R.},
title = {{Design Elements and the Perception of Information Structure}},
booktitle = {Posters IEEE Conf. Information Visualization (InfoVIs)},
year = {2009}
}
@inproceedings{Cawthon2007,
author = {Cawthon, N. and Moere, A. V.},
title = {{The Effect of Aesthetic on the Usability of Data Visualization}},
booktitle = {Proc. IEEE Conf. Information Visualization (InfoVIs)},
doi = {10.1109/IV.2007.147},
pages = {637--648},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4272047},
year = {2007}
}
@article{Ziemkiewicz2009,
author = {Ziemkiewicz, C. and Kosara, R.},
doi = {10.1111/j.1467-8659.2009.01442.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {911--918},
title = {{Preconceptions and Individual Differences in Understanding Visual Metaphors}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2009.01442.x},
volume = {28},
year = {2009}
}
@article{Scholtz2011,
author = {Scholtz, J.},
doi = {10.1177/1473871611407399},
journal = {Information Visualization},
number = {3},
pages = {212--231},
title = {{Developing guidelines for assessing visual analytics environments}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611407399},
volume = {10},
year = {2011}
}
@article{Chen2000,
author = {Chen, C. and Yu, Y.},
doi = {10.1006/ijhc.2000.0422},
journal = {International Journal of ~Human-Computer Studies},
number = {5},
pages = {851-866},
title = {{Empirical studies of information visualization: a meta-analysis}},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904221},
volume = {53},
year = {2000}
}
@inproceedings{Heer2009,
author = {Heer, J. and Kong, N. and Agrawala, M.},
title = {{Sizing the Horizon: The Effects of Chart Size and Layering on the Graphical Perception of Time Series Visualizations}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI)},
pages = {1303--1312},
year = {2009}
}
@article{Simkin2012,
author = {Simkin, D. and Hastie, R.},
journal = {J. American Statistical Association},
number = {398},
pages = {454--465},
title = {{An Information-Processing Analysis of Graph Perception}},
volume = {82},
year = {2012}
}
@article{VonLandesberger2011,
author = {von Landesberger, T. and Kuijper, A. and Schreck, T. and Kohlhammer, J. and van Wijk, J.J. and Fekete, J.-D. and Fellner, D.W.},
doi = {10.1111/j.1467-8659.2011.01898.x},
journal = {Computer Graphics Forum},
number = {6},
pages = {1719--1749},
title = {{Visual Analysis of Large Graphs: State-of-the-Art and Future Research Challenges}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2011.01898.x},
volume = {30},
year = {2011}
}
@article{Ghoniem2005,
author = {Ghoniem, M. and Fekete, J. D. and Castagliola, P.},
doi = {10.1057/palgrave.ivs.9500092},
journal = {Information Visualization},
number = {2},
pages = {114--135},
title = {{On the readability of graphs using node-link and matrix-based representations: a controlled experiment and statistical analysis}},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500092},
volume = {4},
year = {2005}
}
@article{Archambault2010,
author = {Archambault, D. and Purchase, H. and Pinaud, B.},
doi = {10.1109/TVCG.2010.78},
journal = {IEEE transactions on visualization and computer graphics},
number = {4},
pages = {539--552},
title = {{Animation, Small Multiples, and the Effect of Mental Map Preservation in Dynamic Graphs.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/20498503},
volume = {17},
year = {2010}
}
@book{Dang2012,
author = {Dang, L. Q.},
school = {University of British Columbia},
title = {{A functional framework for evaluating visualization applications, with a focus on financial analysis problems}},
year = {2012}
}
@incollection{Smuc2008,
author = {Smuc, M. and Mayr, E. and Lammarsch, T. and Bertone, A. and Aigner, W. and Risku, H. and Miksch, S.},
title = {{Visualizations at First Sight: Do Insights Require Training?}},
booktitle = {USAB, LNCS 5298},
editor = {Holzinger, A.},
pages = {261--280},
publisher = {Springer-Verlag},
year = {2008}
}
@techreport{Saraiya2006a,
author = {Saraiya, P.},
school = {Virginia Polytechnic Institute and State University},
title = {{Insight-Based Studies for Pathway and Microarray Visualization Tools}},
year = {2006}
}
@article{Prabhat2008,
author = {Prabhat and Forsberg, A. and Katzourin, M. and Wharton, K. and Slater, M.},
journal = {IEEE Trans. Visualization and Computer Graphics},
number = {3},
pages = {551--63},
title = {{A comparative study of Desktop, Fishtank, and Cave systems for the exploration of volume rendered confocal data sets.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18461742},
volume = {14},
year = {2008}
}
@inproceedings{Correll2012,
author = {Correll, M. and Albers, D. and Gleicher, M. and Franconeri, S.},
title = {{Comparing Averages in Time Series Data}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
year = {2012}
}
@inproceedings{Metoyer2012,
author = {Metoyer, R. and Lee, B. and Riche, N. H. and Czerwinski, M.},
title = {{Understanding the Verbal Language and Structure of End-User Descriptions of Data Visualizations}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Dunne2012,
author = {Dunne, D. and Riche, N. H. and Lee, B. and Metoyer, R. A. and Robertson, G. G.},
title = {{GraphTrail: Analyzing Large Multivariate and Heterogeneous Networks while Supporting Exploration History}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '12)},
publisher = {ACM},
year = {2012}
}
@article{Pretorius2009,
Author = {Pretorius, A Johannes and Van Wijk, Jarke J},
Journal = {Information Visualization},
Number = {3},
Pages = {153--166},
Title = {What does the user want to see? What do the data want to be?},
Publisher = {Sage},
Volume = {8},
Year = {2009}
}
@inproceedings{Kosara2010,
author = {Kosara, R. and Ziemkiewicz, C.},
title = {{Do Mechanical Turks Dream of Square Pie Charts?}},
booktitle = {Proc. of ~BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV '10)},
year = {2010}
}
@inproceedings{Heer2010,
author = {Heer, J. and Bostock, M.},
title = {{Crowdsourcing graphical perception: using mechanical turk to assess visualization design}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '10)},
pages = {203--212},
url = {http://dl.acm.org/citation.cfm?id=1753357},
year = {2010}
}
@article{Bertini2011a,
author = {Bertini, E. and Lam, H. and Perer, a.},
doi = {10.1177/1473871611421693},
journal = {Information Visualization},
number = {3},
pages = {161--161},
title = {{Summaries: A special issue on Evaluation for Information Visualization}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611421693},
volume = {10},
year = {2011}
}
@article{Goldberg2011,
author = {Goldberg, J. and Helfman, J.},
doi = {10.1177/1473871611406623},
journal = {Information Visualization},
title = {{Eye tracking for visualization evaluation: Reading values on linear versus radial graphs}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611406623},
year = {2011}
}
@article{Conversy2011,
author = {Conversy, S. and Chatty, S. and Hurter, C.},
doi = {10.1177/1473871611415988},
journal = {Information Visualization},
number = {3},
pages = {196--211},
title = {{Visual scanning as a reference framework for interactive representation design}},
url = {http://ivi.sagepub.com/lookup/doi/10.1177/1473871611415988},
volume = {10},
year = {2011}
}
@article{Plaisant2008,
author = {Plaisant, C. and Fekete, J. D. and Grinstein, G.},
title = {{Promoting insight-based evaluation of visualizations: from contest to benchmark repository.}},
doi = {10.1109/TVCG.2007.70412},
journal = {IEEE transactions on visualization and computer graphics},
number = {1},
pages = {120--134},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17993707},
volume = {14},
year = {2008}
}
@inproceedings{Wattenberg2006,
author = {Wattenberg, M.},
title = {{Visual exploration of multivariate graphs}},
booktitle = {Proceedings of the SIGCHI conference on Human Factors in computing systems - CHI '06},
doi = {10.1145/1124772.1124891},
pages = {811--819},
url = {http://portal.acm.org/citation.cfm?doid=1124772.1124891},
year = {2006}
}
@inproceedings{Reilly2007,
author = {Reilly, D. F. and Inkpen, K. M.},
title = {{White rooms and morphing don't mix: setting and the evaluation of visualization techniques}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '07)},
pages = {111--120},
url = {http://dl.acm.org/citation.cfm?id=1240640},
year = {2007}
}
@article{DeOliveira2003,
author = {de Oliveira, M.C.F. and Levkowitz, H.},
title = {{From visual data exploration to visual data mining: A survey}},
doi = {10.1109/TVCG.2003.1207445},
journal = {IEEE transactions on visualization and computer graphics},
month = jul,
number = {3},
pages = {378--394},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1207445},
volume = {9},
year = {2003}
}
@inproceedings{Inbar2007,
author = {Inbar, O. and Tractinsky, N. and Meyer, J.},
title = {{Minimalism in Information Visualization - Attitudes towards Maximizing the Data-Ink Ratio}},
booktitle = {Proc. of the European Conference on Cognitive Ergonomics},
year = {2007}
}
@inproceedings{Bateman2010,
author = {Bateman, S. and Mandryk, R. L. and Gutwin, C. and Genest, A. and McDine, D. and Brooks, C.},
title = {{Useful Junk ? The Effects of Visual Embellishment on Comprehension and Memorability of Charts}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '10)},
year = {2010}
}
@inproceedings{Diakopoulos2011,
author = {Diakopoulos, N. and Kivran-Swaine, F. and Naaman, M.},
title = {{Playable Data: Characterizing the Design Space of Game-y Infographics}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '11)},
year = {2011}
}
@inproceedings{Chi2000,
author = {Chi, E. H.},
title = {{A Taxonomy of Visualization Techniques using the Data State Reference Model}},
booktitle = {Proc. IEEE Symposium on Information Visualization (INFOVIS' 00)},
year = {2000}
}
@inproceedings{Albuquerque2010,
author = {Albuquerque, G. and Eisemann, M. and Lehmann, D. J. and Theisel, H. and Magnor, M.},
title = {{Improving the Visual Analysis of High-dimensional Datasets Using Quality Measures}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology (VAST '10)},
year = {2010}
}
@book{Lam2008b,
author = {Lam, H.},
title = {{Visual Exploratory Analysis of Large Data Sets: Evaluation and Application}},
pages = {293},
publisher= {Univerisity of British Columbia},
year = {2008}
}
@article{Duke2005,
author = {Duke, D. J. and Brodlie, K. W. and Duce, D. A. and Herman, I.},
journal = {IEEE Computer Graphics And Applications},
number = {3},
pages = {6--9},
title = {{Do You See What I Mean?}},
volume = {25},
year = {2005}
}
@article{Eick2005,
author = {Eick, S. G.},
journal = {IEEE computer graphics and applications},
number = {1},
pages = {12--14},
title = {{Information Visualization at 10}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/15691164},
volume = {25},
year = {2005}
}
@inproceedings{Rhyme2004,
author = {Rhyme, T. M. and Hibbard, B. and Johnson, C. and Chen, C. and Eick, S. G.},
title = {{Can We Determine the Top Unresolved Problems of Visualization?}},
booktitle = {Proc. IEEE Conf. Visualization},
pages = {563--565},
year = {2004}
}
@article{Johnson2004,
author = {Johnson, C.},
journal = {IEEE Computer Graphics And Applications},
number = {4},
pages = {13--17},
title = {{Top Scientific Visualization Research Problems}},
volume = {24},
year = {2004}
}
@article{Tory2004a,
author = {Tory, M. and Moller, T.},
doi = {10.1109/TVCG.2004.1260759},
journal = {IEEE transactions on visualization and computer graphics},
number = {1},
pages = {72--84},
title = {{Human Factors in Visualization Research}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/15382699},
volume = {10},
year = {2004}
}
@article{Hibbard1999,
author = {Hibbard, B.},
journal = {PSIGGRAPH Computer Graphics Newsletter},
number = {2},
title = {{Top Ton Visualization Problems}},
volume = {33},
year = {1999}
}
@article{vanWijk2006a,
author = {van Wijk, J. J.},
title = {{Visualization Viewpoints}},
journal = {IEEE Computer Graphics and Applications},
pages = {6--9},
year = {2006}
}
@inproceedings{VanWijk1987,
author = {van Wijk, J. J.},
title = {{The Value of Visualization}},
booktitle = {Proc. IEEE Visualization (VIS '05)},
doi = {10.1109/VISUAL.2005.1532781},
pages = {79--86},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532781},
year = {2005}
}
@article{Kosara2008,
author = {Kosara, R. and Drury, F.},
title = {{Visualization Viewpoints Visualization Criticism}},
journal = {IEEE Computer Graphics And Applications},
number = {3},
pages = {13--15},
volume = {28},
year = {2008}
}
@article{Kerren2008,
author = {Kerren, A. and Stasko, J. T. and Dykes, J.},
title = {{Teaching Information Visualization}},
journal = {Lecture Notes in Computer Science},
pages = {65--91},
volume = {4950},
year = {2008}
}
@article{Slocum2001,
author = {Slocum, T. A. and Blok, C. and Jiang, B. and Koussoulakou, A. and Montello, D. R. and Fuhrmann, S. and Hedley, N. R.},
title = {{Cognitive and Usability Issues in Geovisualization}},
doi = {10.1559/152304001782173998},
journal = {Cartography and Geographic Information Science},
number = {1},
pages = {61--75},
url = {http://openurl.ingenta.com/content/xref?genre=article\&issn=1523-0406\&volume=28\&issue=1\&spage=61},
volume = {28},
year = {2001}
}
@article{Chen2009,
author = {Chen, M. and Ebert, D. and Hagen, H. and Laramee, R. S. and van Liere, R. and Ma, K. and Ribarsky, W. and Scheuermann, G. and Silver, D.},
title = {{Data, Information, and Knowledge in Visualization}},
journal = {IEEE Computer Graphics and Applications },
number = {1},
pages = {12--19},
volume = {29},
year = {2009}
}
@incollection{McCormick1987,
title = {{Visualization in Scientific Computing - A Synopsis}},
booktitle = {Computer Graphics and Applications, IEEE},
doi = {10.1109/MCG.1987.277014},
editor = {McCormick, B. H. and DeFanti, T. A. and Brown, M. D.},
number = {July},
pages = {61 -- 70},
year = {1987}
}
@article{Lam2008a,
author = {Lam, H.},
doi = {10.1109/TVCG.2008.109},
journal = {IEEE transactions on visualization and computer graphics},
title = {{A framework of interaction costs in information visualization.}},
number = {6},
pages = {1149--1156},
url = {http://www.ncbi.nlm.nih.gov/pubmed/18988958},
volume = {14},
year = {2008}
}
@inproceedings{Kang2009,
author = {Kang, Y. A. and Gorg, C. and Stasko, J.},
title = {{Evaluating visual analytics systems for investigative analysis: Deriving design principles from a case study}},
booktitle = {Proc. IEEE Symposium on Visual Analytics Science and Technology},
doi = {10.1109/VAST.2009.5333878},
pages = {139--146},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5333878},
year = {2009}
}
@article{Stasko2008,
author = {Stasko, J. and Gorg, C. and Spence, R.},
title = {{Jigsaw: supporting investigative analysis through interactive visualization}},
doi = {10.1057/palgrave.ivs.9500180},
journal = {Information Visualization},
number = {2},
pages = {118--132},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500180},
volume = {7},
year = {2008}
}
@inproceedings{Faisal2006,
author = {Faisal, S. and Cairns, P. and Blandford, A.},
title = {{Developing user requirements for visualizations of literature knowledge domains}},
booktitle = {Proc. of the 10th International Conference on Information Visualization (IV '06)},
pages = {264--269},
url = {http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=1648271},
year = {2006}
}
@article{Craft2006,
author = {Craft, B. and Cairns, P.},
title = {{Using Sketching to Aid the Collaborative Design of Information Visualisation Software-A Case Study}},
journal = {Human Work Interaction Design: Designing for Human Work},
pages = {103--122},
url = {http://dl.ifip.org/index.php/AICT/article/view/21645},
volume = {221},
year = {2006}
}
@inproceedings{Kobsa2001,
author = {Kobsa, A.},
title = {{An empirical comparison of three commercial information visualization systems}},
booktitle = {Proc. IEEE Symposium on Information Visualization (INFOVIS'01)},
doi = {10.1109/INFVIS.2001.963289},
pages = {123--130},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=963289},
year = {2001}
}
@inproceedings{Mazza2007,
author = {Mazza, R. and Berre, A.},
title = {{Focus Group Methodology for Evaluating Information Visualization Techniques and Tools}},
booktitle = {Proc. of the 1th International Conference on Information Visualization (IV '07)},
doi = {10.1109/IV.2007.51},
pages = {74--80},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4271964},
year = {2007}
}
@incollection{Rester2007,
author = {Rester, M. and Pohl, M. and Wiltner, S. and Hinum, K. and Miksch, S. and Popow, C. and Ohmann, S.},
title = {{Mixing evaluation methods for assessing the utility of an interactive infovis technique}},
booktitle = {Human-Computer Interaction, Part I},
editor = {Jacko, J. A.},
pages = {604--613},
publisher = {HCII},
url = {http://www.springerlink.com/index/j2p615l3j777wp78.pdf},
year = {2007}
}
@article{Pillat2005,
address = {New York, New York, USA},
author = {Pillat, R. M. and Valiati, E. R. A. and Freitas, C.},
title = {{Experimental study on evaluation of multidimensional information visualization techniques}},
doi = {10.1145/1111360.1111363},
journal = {Proc. of the Latin American conference on Human-computer interaction (CLIHC '05)},
pages = {20--30},
url = {http://portal.acm.org/citation.cfm?doid=1111360.1111363},
year = {2005}
}
@inproceedings{Valiati2007,
author = {Valiati, E. R. A. and Pillat, R. M. and Freitas, C. M. D. S. and Pimenta, M. S.},
title = {{Avaliacao Experimental de Classificacao de Tarefas em Visualizacao de Informacoes Multidimensionais}},
booktitle = {Proc. Latin American conference on Human-computer interaction (CLIHC '07) . Workshop of INTERACT '07},
pages = {7--10},
year = {2007}
}
@article{Hetzler2004,
author = {Hetzler, E. and Turner, A.},
title = {{Analysis Experiences Using Information Visualization}},
journal = {IEEE computer graphics and applications},
number = {5},
pages = {22--26},
volume = {24},
year = {2004}
}
@article{Isenberg2011,
author = {Isenberg, P. and Elmqvist, N. and Scholtz, J. and Cernea, D. and Ma, K. L. and Hagen, H.},
title = {{Collaborative visualization: definition, challenges, and research agenda}},
doi = {doi:10.1177/1473871611412817},
journal = {Information Visualization},
number = {4},
pages = {310--326},
url = {http://ivi.sagepub.com/content/10/4/310.short},
volume = {10},
year = {2011}
}
@article{Smuc2009,
author = {Smuc, M. and Mayr, E. and Lammarsch, T. and Aigner, W. and Miksch, S. and Gartner, J.},
title = {{To score or not to score? Tripling insights for participatory design.}},
journal = {IEEE computer graphics and applications},
number = {3},
pages = {29--38},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19642613},
volume = {29},
year = {2009}
}
@article{Mackinlay1986,
author = {Mackinlay, J. D.},
title = {{Automating the design of graphical presentations of relational information}},
doi = {10.1145/22949.22950},
journal = {ACM Transactions on Graphics},
number = {2},
pages = {110--141},
url = {http://portal.acm.org/citation.cfm?doid=22949.22950},
volume = {5},
year = {1986}
}
@inproceedings{Zuk2006a,
author = {Zuk, T. and Carpendale, S.},
title = {{Theoretical Analysis of Uncertainty Visualizations}},
booktitle = {Proc. SPIE \& IS\&T Conf. Electronic Imaging, Visualization and Data Analyses},
volume = {6060},
year = {2006}
}
@article{Stasko2000,
author = {Stasko, J. and Catrambone, R. and Guzdial, M. and ~McDonald, K.},
title = {{An evaluation of space-filling information visualizations for depicting hierarchical structures}},
doi = {10.1006/ijhc.2000.0420},
journal = {International Journal of ~Human-Computer Studies},
number = {5},
pages = {663--694},
url = {http://linkinghub.elsevier.com/retrieve/pii/S1071581900904208},
volume = {53},
year = {2000}
}
@inproceedings{Mark2002,
author = {Mark, G. and Kobsa, A. and Gonzalez, V. M},
title = {{Do four eyes see better than two? Collaborative versus individual discovery in data visualization systems}},
booktitle = {Proc. Sixth International Conference on Information Visualisation},
doi = {10.1109/IV.2002.1028784},
pages = {249--255},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1028784},
year = {2002}
}
@inproceedings{Perer2008a,
author = {Perer, A. and Shneiderman, B.},
title = {{Systematic yet flexible discovery: guiding domain experts through exploratory data analysis}},
booktitle = {Proc. of the 13th international conference on Intelligent user interfaces (IUI '08)},
pages = {109--118},
url = {http://portal.acm.org/citation.cfm?id=1378788},
year = {2008}
}
@inproceedings{Perer2008,
author = {Perer, A. and Shneiderman, B.},
title = {{Integrating statistics and visualization: case studies of gaining clarity during exploratory data analysis}},
booktitle = {Proc. of the twenty-sixth annual SIGCHI conference on Human factors in computing systems (CHI '08)},
pages = {265--274},
url = {http://portal.acm.org/citation.cfm?id=1357101},
year = {2008}
}
@article{Saraiya2005,
title = {{Visualizing biological pathways: requirements analysis, systems evaluation and research agenda}},
author = {Saraiya, P. and North, C. and Duca, K.},
doi = {10.1057/palgrave.ivs.9500102},
journal = {Information Visualization},
number = {3},
pages = {191--205},
url = {http://ivi.sagepub.com/lookup/doi/10.1057/palgrave.ivs.9500102},
volume = {4},
year = {2005}
}
@inproceedings{Juarez-Espinosa2003,
author = {Juarez-Espinosa, O.},
title = {{CAEVA: cognitive architecture to evaluate visualization applications}},
booktitle = {Proc. on Seventh International Conference on Information Visualization (IV '03)},
doi = {10.1109/IV.2003.1218045},
pages = {589--594},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1218045},
year = {2003}
}
@article{Freitas2002,
title = {{Evaluating Usability of Information Visualization Techniques}},
author = {Freitas, C. and Luzzardi, P. R. G. and Cava, R. A and Pimenta, M. S. and Winckler, M. A. and Nedel, L.},
journal = {Proc. Advanced Visual Interfaces (AVI '02)},
pages = {373--374},
url = {http://www.irit.fr/~Marco.Winckler/projects/spider/publications/IHC2002-FreitasEtAl.pdf},
year = {2002}
}
@article{Mark2005,
author = {Mark, G. and Kobsa, A.},
title = {{The effects of collaboration and system transparency on CIVE usage: an empirical study and model}},
journal = {Presence: Teleoperators \& Virtual Environments},
number = {1},
pages = {60--80},
publisher = {MIT Press},
url = {http://www.mitpressjournals.org/doi/abs/10.1162/1054746053890279},
volume = {14},
year = {2005}
}
@article{Trainer2008,
author = {Trainer, E. and Quirk, S. and de Souza, C. and Redmiles, D.},
doi = {10.1109/VLHCC.2008.4639062},
journal = {2008 IEEE Symposium on Visual Languages and ~Human-Centric Computing},
pages = {78--81},
publisher = {Ieee},
title = {{Analyzing a socio-technical visualization tool using usability inspection methods}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4639062},
year = {2008}
}
@article{Cockburn2000,
author = {Cockburn, A. and ~McKenzie, B.},
title = {{An evaluation of cone trees}},
journal = {People and Computers: Usability or Else. British Computer Society Conf. on Human Computer Interaction},
pages = {425-436},
publisher = {Citeseer},
url = {http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.34.8987\&rep=rep1\&type=pdf},
volume = {XIV},
year = {2000}
}
@inproceedings{Cockburn2001,
author = {Cockburn, A. and ~McKenzie, B.},
title = {{3D or not 3D? Evaluating the effect of the third dimension in a document management system}},
booktitle = {Proc. of the SIGCHI conference on Human factors in computing systems (CHI '01)},
pages = {434-441},
url = {http://portal.acm.org/citation.cfm?id=365309},
year = {2001}
}
@article{Munzner2009,
author = {Munzner, T.},
title = {{A nested model for visualization design and validation.}},
doi = {10.1109/TVCG.2009.111},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {921--8},
url = {http://www.ncbi.nlm.nih.gov/pubmed/19834155},
volume = {15},
year = {2009}
}
@article{Wattenberg2005,
author = {Wattenberg, M.},
title = {{Baby names, visualization, and social data analysis}},
doi = {10.1109/INFVIS.2005.1532122},
journal = {IEEE Symposium on Information Visualization (INFOVIS '05).},
pages = {1--7},
publisher = {Ieee},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1532122},
year = {2005}
}
@article{Tory2007,
author = {Tory, M. and Sprague, D. and Wu, F. and So, W. Y. and Munzner, T.},
doi = {10.1109/TVCG.2007.70596},
title = {{Spatialization design: comparing points and landscapes.}},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {1262--9},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968073},
volume = {13},
year = {2007}
}
@article{Pousman2007,
author = {Pousman, Z. and Stasko, J. and Mateas, M.},
doi = {10.1109/TVCG.2007.70541},
title = {{Casual information visualization: depictions of data in everyday life.}},
journal = {IEEE transactions on visualization and computer graphics},
number = {6},
pages = {1145--52},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17968058},
volume = {13},
year = {2007}
}
@article{Yost2006,
author = {Yost, B. and North, S. C.},
doi = {10.1109/TVCG.2006.184},
journal = {IEEE transactions on visualization and computer graphics},
number = {5},
pages = {837-844},
title = {{The perceptual scalability of visualization.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/17080807},
volume = {12},
year = {2006}
}
@article{Cockburn2008a,
author = {Cockburn, A. and Karlson, A.},
doi = {10.1145/1456650.145662},
journal = {ACM Computing Surveys},
pages = {1-42},
title = {{A review of overview+ detail, zooming, and focus+ context interfaces}},
url = {http://portal.acm.org/citation.cfm?id=1456652\&dl=ACM},
volume = {41},
year = {2008}
}
@inproceedings{Lam2008,
author = {Lam, H. and Munzner, T.},
title = {{Increasing the utility of quantitative empirical studies for meta-analysis}},
booktitle = {Proc. of the AVI workshop on ~BEyond time and errors: novel evaluation methods for information visualization (BELIV '08)},
doi = {10.1145/1377966.1377969},
url = {http://portal.acm.org/citation.cfm?doid=1377966.1377969},
year = {2008}
}
@techreport{Chen2013,
author = {Chen, M. and Floridi, L. and Borgo, R.},
title = {{What is visualization really for?}},
institution = {University of Oxford},
pages = {1--11},
url = {arXiv:1305.5670},
year = {2013}
}
@phdthesis{Borkin2014,
author = {Borkin, M. A.},
title = {{Perception, Cognition, and Effectiveness of Visualizations with Applications in Science and Engineering}},
school = {Harvard University},
year = {2014}
}
}}}
{{{
@article{Booshehrian2012,
author = {Booshehrian, M. and M\"{o}ller, T. and Peterman, R. M. and Munzner, T.},
doi = {10.1111/j.1467-8659.2012.03116.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {1235--1244},
title = {{Vismon: Facilitating analysis of trade-offs, uncertainty, and sensitivity In fisheries management decision making}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2012.03116.x},
volume = {31},
year = {2012}
}
@techreport{Booshehrian2011,
author = {Booshehrian, M. and Moller, T. and Peterman, R. M. and Munzner, T.},
institution = {TR 2011-04. School of Computing Science, Simon Fraser University},
pages = {1--16},
title = {{Vismon: Facilitating Risk Assessment and Decision Making In Fisheries Management}},
year = {2011}
}
@article{Torsney-Weir2011,
author = {Torsney-Weir, T. and Saad, A. and Moller, T. and Weber, B. and Hege, H. C. and Verbavatz, J. M. and Bergner, Steven},
doi = {10.1109/TVCG.2011.248},
journal = {IEEE transactions on visualization and computer graphics},
number = {12},
pages = {1892--1901},
title = {{Tuner: principled parameter finding for image segmentation algorithms using visual response surface exploration.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034306},
volume = {17},
year = {2011}
}
@article{Pretorius2011,
author = {Pretorius, A. J. and Bray, M. A. P. and Carpenter, A. E. and Ruddle, R. A.},
doi = {10.1109/TVCG.2011.253},
journal = {IEEE transactions on visualization and computer graphics},
number = {12},
pages = {2402--2411},
title = {{Visualization of parameter space for image analysis.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034361},
volume = {17},
year = {2011}
}
@article{Eskelinen2008,
author = {Eskelinen, P. and Miettinen, K. and Klamroth, K. and Hakanen, J.},
doi = {10.1007/s00291-008-0151-6},
journal = {OR Spectrum},
number = {1},
pages = {211--227},
title = {{Pareto navigator for interactive nonlinear multiobjective optimization}},
url = {http://www.springerlink.com/index/10.1007/s00291-008-0151-6},
volume = {32},
year = {2008}
}
@article{Piringer2010a,
author = {Piringer, H. and Berger, W. and Krasser, J.},
doi = {10.1111/j.1467-8659.2009.01684.x},
journal = {Computer Graphics Forum},
number = {3},
pages = {983--992},
title = {{HyperMoVal: Interactive Visual Validation of Regression Models for Real-Time Simulation}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2009.01684.x},
volume = {29},
year = {2010}
}
@article{Berger2011,
author = {Berger, W. and Piringer, H. and Filzmoser, P. and Gr\"{o}ller, E.},
doi = {10.1111/j.1467-8659.2011.01940.x},
issn = {01677055},
journal = {Computer Graphics Forum},
month = jun,
number = {3},
pages = {911--920},
title = {{Uncertainty-Aware Exploration of Continuous Parameter Spaces Using Multivariate Prediction}},
url = {http://doi.wiley.com/10.1111/j.1467-8659.2011.01940.x},
volume = {30},
year = {2011}
}
@article{Holt2011,
author = {Holt, K. R. and Peterman, R. M. and Cox, S. P.},
doi = {10.1139/F2011-022},
journal = {Canadian Journal of Fisheries and Aquatic Sciences},
number = {68},
pages = {880--897},
title = {{Trade-offs between monitoring objectives and monitoring effort when classifying regional conservation status of Pacific salmon (Oncorhynchus spp.) populations}},
volume = {897},
year = {2011}
}
@incollection{Peterman2009,
author = {Peterman, R. M.},
booktitle = {The Future of Fisheries Science in North America},
pages = {167--184},
title = {{Fisheries Science in the Future}},
year = {2009}
}
@article{Peterman2004b,
author = {Peterman, R. M.},
doi = {10.1016/j.icesjms.2004.08.017},
journal = {ICES Journal of Marine Science},
month = dec,
number = {8},
pages = {1331--1343},
title = {{Possible solutions to some challenges facing fisheries scientists and managers}},
url = {http://icesjms.oxfordjournals.org/cgi/doi/10.1016/j.icesjms.2004.08.017},
volume = {61},
year = {2004}
}
@article{Peterman1975,
author = {Peterman, R. M.},
journal = {Fish. Red. Board Can.},
pages = {2179--2188},
title = {{New Techniques for Policy Evaluation in Ecological Systems: Methodology for a Case Study of Pacific Salmon Fisheries}},
volume = {32},
year = {1975}
}
}}}
4 pm Statistics Department seminar ESB 1012
*[[Slides|http://www.stat.ubc.ca/lib/FCKuserfiles//HadleyWickhambigvis-1.pdf]]
>//R has a notorious reputation for not being able to deal with "big" data (and ggplot2 is a frequent culprit). Fortunately, this isn't an underlying problem with R, and it's something that we can fix with good programming practices and intelligent use of compiled code. In this talk, I'll introduce a new package, bigvis, that aims to make it easier (and faster) to work with very large datasets. My motivating example, used throughout the talk, I will be a data set of flight delay information for about 76 million flights in the continental US.//
*[[bigvis|https://github.com/hadley/bigvis]] R package
!!Motivation
*With 100M pts, you can't use data frames. Use vectors instead.
*Data cleaning. There's always bad data. In human-generated datasets, expect 2% (2% of 76M is still a lot)
*slow to plot, giant hairball issues; bottlenecks: pixels on screen (3K pts in 1D, 3M pts in 2D), memory issues.
!!Data
*76 million flights in the continental US (2001-2009) from [[ASA data expo 2009|http://stat-computing.org/dataexpo/2009/the-data.html]]
!!Process
*''Condense'' (Bin and summarize)
**''Bin'': use fixed, not variable-sized bins (variable size is hard), select your own bin widths
**''Summarize'':
{{{
dist_s <- condense(bin(dist,10))
autoplot(dist_s)
}}}
**[[bigvis|https://github.com/hadley/bigvis]] always shows you missing values, {{{na.rm}}} removes missing values
**On binning values: //"humans really like the number 5"//
**Jim Gray on categories of statistics:
***//distributive//: e.g. sum, count, min, max (1 value needed); re-aggregation / re-binning is easy, trivially parellizable
***//algebraic//: e.g. mean, skew (m values needed); re-aggregation is easy, trivially parellizable
***//holistic//: e.g. median, quantile, cardinality; re-aggregation is hard, not parellizable
**However, if you can fit your data in memory, even computing the median is quick
*''Smooth'': methods in order of increasing sophistication
**//naive binning//: bin's mean value
**//running mean//: mean of nearest M points
**//kernel mean//: {{{ k(x) = (1 - [x]}}}^^3^^{{{)}}}^^2^^{{{I}}}~~[ x ]~~{{{ < 1}}} (convolution, ~Nadayara-Watson regression)
**//kernel regression//
**//kernel robust regression// (loess curve): highly resistant to outliers
**what's the best bandwidth for smoothing? Not a trivial provlem. rmse, leave-one-out cross-validation
*''Visualize'' with [[bigvis|https://github.com/hadley/bigvis]]
**challenges: outliers, showing the display count, showing the number of missing values
**dealing with outliers: remove small percentages of observations from the convex hull of the data (also use the Box-Cox transformation for positive values, Modulus transformation otherwise)
**[[rcpp|https://github.com/hadley/devtools/wiki/Rcpp]] (Eddelbuettel, Francois, Allaire): get your C++ code to run in R; higher performance, precise memory allocation and copying, much easier to connect to R than C or Fortran, not hard to learn (you'll only be writing tens-hundeds of lines of C++, not thousands to tens of thousands - all the boilerplate code, compilation and linking is handled by rcpp) - see the [[rcpp gallery|http://gallery.rcpp.org/]]
**[[shiny|https://github.com/rstudio/shiny/]] (Chen, Chang) - interactive webapps in R, no HTML, JS, CSS required
***ui.r and server.r, shiny makes the connections; deploy locally or on [[shiny server|http://www.rstudio.com/shiny/]]
***[[Shiny: TV show rankings|http://glimmer.rstudio.com/pssguy/TVShowRatings/]] ([[blog post|http://www.premiersoccerstats.com/wordpress/?p=1380]], [[gist|https://gist.github.com/pssguy/5498431]])
***[[Shiny: Aggregate Dialect Difference|http://spark.rstudio.com/jkatz/DialectMap/]]
!!Q&A
*''Q'': Competition to [[bigvis|https://github.com/hadley/bigvis]]?
**''A'': [[imMens|http://vis.stanford.edu/papers/immens]] via Stanford vis group
{{{
@inproceedings{Tomlinson2012a,
author = {Tomlinson, B. and Silberman, M. S. and Patterson, D. and Pan, Y. and Blevis, E.},
title = {{Collapse Informatics: Augmenting the Sustainability \& ICT4D Discourse in HCI}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {655--664},
year = {2012}
}
@inproceedings{Antin2012,
author = {Antin, J. and Shaw, A.},
title = {Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012},
pages = {2925--2934},
url = {http://doi.acm.org/10.1145/2207676.2208699},
}
@inproceedings{Tanenbaum2012,
author = {Tanenbaum, J. and Tanenbaum, K. and Wakkary, R.},
title = {Steampunk as design fiction},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012},
pages = {1583--1592},
url = {http://doi.acm.org/10.1145/2208276.2208279},
}
@inproceedings{Dell2012,
author = {Dell, N. and Vaidyanathan, V. and Medhi, I. and Cutrell, E. and Thies, W.},
title = {{"Yours is Better" Participant Response Bias in HCI}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1321--1330},
year = {2012}
}
@inproceedings{Friess2012,
author = {Friess, E.},
title = {{Personas and Decision Making in the Design Process: An Ethnographic Case Study}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1209--1218},
year = {2012}
}
@inproceedings{Nagel2012,
author = {Nagel, T. and Duval, E. and Moere, A. V.},
title = {{Interactive Exploration of Geospatial Network Visualization}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {557--572},
year = {2012}
}
@inproceedings{Lee2012,
author = {Lee, H. and Lee, S. and Kim, N. and Seo, J.},
title = {{JigsawMap: Connecting the Past to the Future by Mapping Historical Textual Cadasters}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {463--472},
year = {2012}
}
@article{Hayes2011,
author = {Hayes, Gillian R.},
doi = {10.1145/1993060.1993065},
journal = {ACM Transactions on Computer-Human Interaction},
number = {3},
pages = {1--20},
title = {{The relationship of action research to human-computer interaction}},
url = {http://dl.acm.org/citation.cfm?doid=1993060.1993065},
volume = {18},
year = {2011}
}
@inproceedings{Endert2012,
author = {Endert, A. and Fiaux, P. and North, C.},
title = {{Semantic Interaction for Visual Text Analytics}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Davies2012,
author = {Davies, T. and Beeharee, A. K.},
title = {{The Case of the Missed Icon: Change Blindness on Mobile Devices}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1451--1460},
year = {2012}
}
@inproceedings{Fisher2012,
author = {Fisher, K. and Counts, S. and Kittur, A. and Ave, F.},
title = {{Distributed Sensemaking: Improving Sensemaking by Leveraging the Efforts of Previous Users}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Ziemkiewicz2012,
author = {Ziemkiewicz, C. and Gomez, S. and Laidlaw, D. H.},
title = {{Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Fisher2012a,
author = {Fisher, D. and Popov, I. and Drucker, S. M. and Schraefel, Mc},
title = {{Trust Me, I'm Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Chuang2012,
author = {Chuang, J. and Ramage, D. and Manning, C. D. and Heer, J.},
title = {{Interpretation and Trust: Designing Model-Driven Visualizations for Text Analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Correll2012,
author = {Correll, M. and Albers, D. and Gleicher, M. and Franconeri, S.},
title = {{Comparing Averages in Time Series Data}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Metoyer2012,
author = {Metoyer, R. and Lee, B. and Riche, N. H. and Czerwinski, M.},
title = {{Understanding the Verbal Language and Structure of End-User Descriptions of Data Visualizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Willett2012,
author = {Willett, W. and Heer, J. and Agrawala, M.},
title = {{Strategies for Crowdsourcing Social Data Analysis}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
publisher = {ACM},
year = {2012}
}
@inproceedings{Jansen2012,
author = {Jansen, Y. and Dragicevic, P. and Fekete, J. D.},
title = {{Tangible remote controllers for wall-size displays}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208691},
pages = {2865},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208691},
year = {2012}
}
@inproceedings{Diakopoulos2012,
author = {Diakopoulos, N. and {De Choudhury}, M. and Naaman, M.},
title = {{Finding and assessing social media information sources in the context of journalism}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208409},
pages = {2451},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208409},
year = {2012}
}
@inproceedings{Dunne2012a,
author = {Dunne, C. and Riche, N. H. and Lee, B. and Metoyer, R. A. and Robertson, G. G.},
title = {{GraphTrail: Analyzing Large Multivariate, Heterogeneous Networks while Supporting Exploration History}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1663--1672},
year = {2012}
}
@inproceedings{Thudt2012,
author = {Thudt, A. and Hinrichs, U. and Carpendale, S.},
title = {{The Bohemian Bookshelf: Supporting Serendipitous Book Discoveries through Information Visualization}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1461--1470},
year = {2012}
}
@inproceedings{Kong2012,
author = {Kong, N. and Grossman, T. and Hartmann, B. and Fitzmaurice, G. and Agrawala, M.},
title = {{Delta: A Tool For Representing and Comparing Workflows}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1027--1036},
year = {2012}
}
@inproceedings{Heer2012,
author = {Heer, J. and Stone, M.},
title = {{Color naming models for color selection, image editing and palette design}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208547},
pages = {1007},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208547},
year = {2012}
}
@inproceedings{Faste2012,
author = {Faste, H. and Lin, H.},
title = {{The untapped promise of digital mind maps}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208548},
pages = {1017},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208548},
year = {2012}
}
@inproceedings{Pierce2012,
author = {Pierce, J.},
title = {{Undesigning Technology: Considering the Negation of Design by Design}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {957--966},
year = {2012}
}
@inproceedings{Rode2012,
author = {Rode, J. A. and Blythe, M. and Nardi, B.},
title = {{Qualitative Research in HCI}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2803--2806},
year = {2012}
}
@inproceedings{Swearngin2012,
author = {Swearngin, A. and Cohen, M. and John, B. and Bellamy, R.},
title = {{Easing the generation of predictive human performance models from legacy systems}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208415},
pages = {2489},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208415},
year = {2012}
}
@inproceedings{Karapanos2012,
author = {Karapanos, E. and Jain, J. and Hassenzahl, M.},
title = {{Theories, Methods and Case Studies of Longitudinal HCI Research}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2727--2730},
year = {2012}
}
@inproceedings{Obrist2012,
author = {Obrist, M. and ~Vaananen-Vainio-Mattila, K. and Roto, V. and Vermeeren, A. and Law, E. L. and Buie, E.},
title = {{Theories behind UX Research and How They Are Used in Practice}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2751--2754},
year = {2012}
}
@inproceedings{Gomez2012,
author = {Gomez, S. and Laidlaw, D.},
title = {{Modeling task performance for a crowd of users from interaction histories}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208412},
pages = {2465},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208412},
year = {2012}
}
@inproceedings{Teo2012,
author = {Teo, L. H. and John, B. E. and Blackmon, M. H.},
title = {{CogTool-Explorer: A Model of Goal-Directed User Exploration that Considers Information Layout}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2479--2488},
year = {2012}
}
@inproceedings{Abeele2012,
author = {Abeele, V. V. and Hauters, E. and Zaman, B.},
title = {{Increasing the Reliability and Validity of Quantitative Laddering Data with LadderUX}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2057--2062},
year = {2012}
}
@inproceedings{Kusunoki2012,
author = {Kusunoki, D.},
title = {{Applying Participatory Design Theory to Designing Evaluation Methods}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1895--1900},
year = {2012}
}
@inproceedings{Obrist2012a,
author = {Obrist, M. and ~Vaananen-Vainio-Mattila, K. and Roto, V. and Law, E. L. and Vermeeren, A. and Kuutti, K.},
title = {{In Search of Theoretical Foundations for UX Research and Practice}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2012}
}
@inproceedings{Kaptein2012,
author = {Kaptein, M. and Robertson, J.},
title = {{Rethinking statistical analysis methods for CHI}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
doi = {10.1145/2207676.2208557},
pages = {1105},
url = {http://dl.acm.org/citation.cfm?doid=2207676.2208557},
year = {2012}
}
@inproceedings{Jain2012,
author = {Jain, J. and Boyce, S.},
title = {{Case Study: Longitudinal Comparative Analysis for Analyzing User Behavior}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {793--799},
year = {2012}
}
@inproceedings{Switzky2012,
author = {Stitzky, A.},
title = {{Incorporating UCD Into the Software Development Process: A Case Study}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {469--483},
year = {2012}
}
@inproceedings{Lingel2012,
author = {Lingel, J.},
title = {{Ethics and Dilemmas of Online Ethnography}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {41--50},
year = {2012}
}
@inproceedings{Gill2012,
author = {Gill, Z.},
title = {{User-Driven Collaborative Intelligence Social Networks as Crowdsourcing Ecosystems}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {161--170},
year = {2012}
}
@inproceedings{Ko2012,
author = {Ko, A. J.},
title = {{Mining Whining in Support Forums with Frictionary}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {191--200},
year = {2012}
}
@inproceedings{Jianu2012,
author = {Jianu, R. and Laidlaw, D.},
title = {{An Evaluation of How Small User Interface Changes Can Improve Scientists' Analytic Strategies}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2953--2962},
year = {2012}
}
@inproceedings{Eisenstein2012,
author = {Eisenstein, J. and Chau, D. H. P. and Kittur, A. and Xing, E. P.},
title = {{TopicViz: Interactive Topic Exploration in Document Collections}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2177--2182},
year = {2012}
}
@inproceedings{Muralidharan2012,
author = {Muralidharan, A. and Hearst, M.},
title = {{A Sensemaking Environment for Literature Study}},
booktitle = {Proc. Extended Abstracts ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {1955--1960},
year = {2012}
}
}}}
{{{
@inproceedings{Polowinski2013,
author = {Polowinski, J. and Voigt, M.},
title = {VISO: a shared, formal knowledge base as a foundation for semi-automatic infovis systems},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {1791--1796},
url = {http://doi.acm.org/10.1145/2468356.2468677},
doi = {10.1145/2468356.2468677},
}
@inproceedings{Perer2013,
author = {Perer, A. and Gotz, D.},
title = {Data-driven exploration of care plans for patients},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {439--444},
url = {http://doi.acm.org/10.1145/2468356.2468434},
doi = {10.1145/2468356.2468434},
}
@inproceedings{Dork2013a,
author = {Dͮ and Feng, P. and Collins, C. and Carpendale, S.},
title = {Critical InfoVis: exploring the politics of visualization},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2189--2198},
url = {http://doi.acm.org/10.1145/2468356.2468739},
doi = {10.1145/2468356.2468739},
}
@inproceedings{Kirman2013,
author = {Kirman, B. and Linehan, C. and Lawson, S. and O'Hara, D.},
title = {CHI and the future robot enslavement of humankind: a retrospective},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2199--2208},
url = {http://doi.acm.org/10.1145/2468356.2468740},
doi = {10.1145/2468356.2468740},
}
@inproceedings{Wallace2013,
author = {Wallace, J. R. and Scott, S. D. and MacGregor, C. G.},
title = {Collaborative sensemaking on a digital tabletop and personal tablets: prioritization, comparisons, and tableaux},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {3345--3354},
url = {http://doi.acm.org/10.1145/2470654.2466458},
doi = {10.1145/2470654.2466458},
}
@inproceedings{Yang2013,
author = {Yang, H. and Pupons-Wickham, D. and Chiticariu, L. and Li, Y. and Nguyen, B. and Carreno-Fuentes, A.},
title = {I can do text analytics!: designing development tools for novice developers},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {1599--1608},
url = {http://doi.acm.org/10.1145/2466110.2466212},
doi = {10.1145/2466110.2466212},
}
@inproceedings{Alper2013,
author = {Alper, B. and Bach, B. and Riche, N. H. and Isenberg, T. and Fekete, J. D.},
title = {Weighted graph comparison techniques for brain connectivity analysis},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {483--492},
url = {http://doi.acm.org/10.1145/2470654.2470724},
doi = {10.1145/2470654.2470724},
}
@inproceedings{Kim2013,
author = {Kim, S. and Paulos, E. and Mankoff, J.},
title = {inAir: a longitudinal study of indoor air quality measurements and visualizations},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2745--2754},
url = {http://doi.acm.org/10.1145/2470654.2481380},
doi = {10.1145/2470654.2481380},
}
@inproceedings{Jansen2013,
author = {Jansen, Y. and Dragicevic, P. and Fekete, J. D.},
title = {Evaluating the efficiency of physical visualizations},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2593--2602},
url = {http://doi.acm.org/10.1145/2470654.2481359},
doi = {10.1145/2470654.2481359},
}
@inproceedings{Jones2013,
author = {Jones, B. R. and Benko, H. and Ofek, E. and Wilson, A. D.},
title = {IllumiRoom: peripheral projected illusions for interactive experiences},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {869--878},
url = {http://doi.acm.org/10.1145/2470654.2466112},
doi = {10.1145/2470654.2466112},
}
@inproceedings{Toker2013,
author = {Toker, D. and Conati, C. and Steichen, B. and Carenini, G.},
title = {Individual user characteristics and information visualization: connecting the dots through eye tracking},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {295--304},
url = {http://doi.acm.org/10.1145/2470654.2470696},
doi = {10.1145/2470654.2470696},
}
@inproceedings{Dork2013,
author = {Dͮ and Lam, H. and Benjelloun, O.},
title = {Accentuating visualization parameters to guide exploration},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {1755--1760},
url = {http://doi.acm.org/10.1145/2468356.2468671},
doi = {10.1145/2468356.2468671}
}
@inproceedings{Munteanu2013,
author = {Munteanu, C. and Fournier, H. and Lapointe, J. F. and Emond, B. and Kondratova, I.},
title = {We'll take it from here: letting the users take charge of the evaluation and why that turned out well},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2383--2384},
url = {http://doi.acm.org/10.1145/2468356.2468778},
doi = {10.1145/2468356.2468778}
}
@inproceedings{Trimble2013,
author = {Trimble, J. and Dayton, T. and Crocker, A.},
title = {The democratization of mission control: empowering users},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2381--2382},
numpages = {2},
url = {http://doi.acm.org/10.1145/2468356.2468777},
doi = {10.1145/2468356.2468777}
}
@inproceedings{Wilfinger2013,
author = {Wilfinger, D. and Meschtscherjakov, A. and Perterer, N. and Murer, M. and Laminger, A. and Tscheligi, M.},
title = {Automotive HMI test package: an exploitable approach to study in-car HMIs},
booktitle = {Proc. Extended Abstracts of ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2377--2380},
url = {http://doi.acm.org/10.1145/2468356.2468776},
doi = {10.1145/2468356.2468776}
}
@article{Akers2012,
author = {Akers, D. and Jeffries, R. and Simpson, M. and Winograd, T.},
title = {Backtracking events as indicators of usability problems in creation-oriented applications},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
volume = {19},
number = {2},
year = {2012},
pages = {16:1--16:40},
url = {http://doi.acm.org/10.1145/2240156.2240164},
doi = {10.1145/2240156.2240164}
}
@inproceedings{Matejka2013,
author = {Matejka, J. and Grossman, T. and Fitzmaurice, G.},
title = {Patina: dynamic heatmaps for visualizing application usage},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {3227--3236},
url = {http://doi.acm.org/10.1145/2466416.2466442},
doi = {10.1145/2466416.2466442}
}
@inproceedings{Agapie2013,
author = {Agapie, E. and Golovchinsky, G. and Qvarfordt, P.},
title = {Leading people to longer queries},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {3019--3022},
url = {http://doi.acm.org/10.1145/2470654.2481418},
doi = {10.1145/2470654.2481418}
}
@inproceedings{Zhao2013,
author = {Zhao, J. and Wigdor, D. and Balakrishnan, R.},
title = {TrailMap: facilitating information seeking in a multi-scale digital map via implicit bookmarking},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {3009--3018},
numpages = {10},
url = {http://doi.acm.org/10.1145/2470654.2481417},
doi = {10.1145/2470654.2481417}
}
@inproceedings{Feild2013,
author = {Feild, H. and White, R. W. and Fu, X.},
title = {Supporting orientation during search result examination},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2999--3008},
url = {http://doi.acm.org/10.1145/2470654.2481416},
doi = {10.1145/2470654.2481416}
}
@inproceedings{Goyal2013,
author = {Goyal, N. and Leshed, G. and Fussell, S. R.},
title = {Effects of visualization and note-taking on sensemaking and analysis},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2721--2724},
url = {http://doi.acm.org/10.1145/2470654.2481376},
doi = {10.1145/2470654.2481376}
}
@inproceedings{Wang2013,
author = {Wang, Y. and Echenique, A. and Shelton, M. and Mark, G.},
title = {A comparative evaluation of multiple chat stream interfaces for information-intensive environments},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2717--2720},
url = {http://doi.acm.org/10.1145/2470654.2481375},
doi = {10.1145/2470654.2481375}
}
@inproceedings{Liao2013,
author = {Liao, Q. V. and Fu, W. T.},
title = {Beyond the filter bubble: interactive effects of perceived threat and topic involvement on selective exposure to information},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2359--2368},
url = {http://doi.acm.org/10.1145/2470654.2481326},
doi = {10.1145/2470654.2481326}
}
@inproceedings{Monroe2013,
author = {Monroe, M. and Lan, R. and Morales del Olmo, J. and Shneiderman, B. and Plaisant, C. and Millstein, J.},
title = {The challenges of specifying intervals and absences in temporal queries: a graphical language approach},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2349--2358},
url = {http://doi.acm.org/10.1145/2470654.2481325},
doi = {10.1145/2470654.2481325}
}
@inproceedings{Sko2013,
author = {Sko, T. and Gardner, H. J. and Martin, M.},
title = {Non-parametric decision trees and online HCI},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2103--2106},
url = {http://doi.acm.org/10.1145/2470654.2481288},
doi = {10.1145/2470654.2481288}
}
@inproceedings{Lewis2013,
author = {Lewis, J. R. and Utesch, B. S. and Maher, D. E.},
title = {UMUX-LITE: when there's no time for the SUS},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2099--2102},
url = {http://doi.acm.org/10.1145/2470654.2481287},
doi = {10.1145/2470654.2481287}
}
@inproceedings{Hedegaard2013,
author = {Hedegaard, S. and Simonsen, J. G.},
title = {Extracting usability and user experience information from online user reviews},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2089--2098},
url = {http://doi.acm.org/10.1145/2470654.2481286},
doi = {10.1145/2470654.2481286}
}
@inproceedings{Drucker2013,
author = {Drucker, S. M. and Fisher, D. and Sadana, R. and Herron, J. and schraefel, m.c.},
title = {TouchViz: a case study comparing two interfaces for data analytics on tablets},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {2301--2310},
url = {http://doi.acm.org/10.1145/2470654.2481318},
doi = {10.1145/2470654.2481318}
}
@inproceedings{Erickson2013,
author = {Erickson, T. and Li, M. and Kim, Y. and Deshpande, A. and Sahu, S. and Chao, T. and Sukaviriya, P. and Naphade, M.},
title = {The dubuque electricity portal: evaluation of a city-scale residential electricity consumption feedback system},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
year = {2013},
pages = {1203--1212},
url = {http://doi.acm.org/10.1145/2470654.2466155},
doi = {10.1145/2470654.2466155}
}
@inproceedings{Hullman2013,
author = {Hullman, J. and Diakopoulos, N. and Adar, E.},
title = {{Contextifier: Automatic generation of annotated stock visualizations}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2707--2716},
url = {http://doi.acm.org/10.1145/2470654.2481374},
doi = {10.1145/2470654.2481374},
year = {2013}
}
@inproceedings{Fitchett2013,
author = {Fitchett, S. and Cockburn, A. and Gutwin, C.},
title = {{Improving navigation-based file retrieval}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2329--2338},
url = {http://doi.acm.org/10.1145/2470654.2481323},
doi = {10.1145/2470654.2481323},
year = {2013}
}
@inproceedings{Dunne2013,
author = {Dunne, C. and Shneiderman, B.},
title = {{Motif simplification: Improving network visualization readability with fan, connector, and clique glyphs}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {3247--3256},
url = {http://doi.acm.org/10.1145/2470654.2466444},
doi = {10.1145/2470654.2466444},
year = {2013}
}
@inproceedings{Correll2013,
author = {Correll, M. A. and Alexander, E..C. and Gleicher, M.},
title = {{Quantity estimation in visualizations of tagged text}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2697--2706},
url = {http://doi.acm.org/10.1145/2470654.2481373},
doi = {10.1145/2470654.2481373},
year = {2013}
}
@inproceedings{Kleek2013,
author = {Kleek, M. v. and Smith, D. A. and Packer, H. S. and Skinner, J. and Shadbolt, N. R.},
title = {{Carp䡺 Supporting serendipitous data integration in personal information management}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2339--2348},
url = {http://doi.acm.org/10.1145/2470654.2481324},
doi = {10.1145/2470654.2481324},
year = {2013}
}
@inproceedings{Perin2013,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Interactive horizon graphs: Improving the compact visualization of multiple time series}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
url = {http://doi.acm.org/10.1145/2470654.2466441},
doi = {10.1145/2470654.2466441},
pages = {3217--3226},
year = {2013}
}
@inproceedings{Fuchs2013,
author = {Fuchs, J. and Fischer, F. and Mansmann, F. and Bertini, E. and Isenberg, P.},
title = {{Evaluation of alternative glyph designs for time series data in a small multiple setting}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {3237--3246},
url = {http://doi.acm.org/10.1145/2470654.2466443},
doi = {10.1145/2470654.2466443},
year = {2013}
}
@inproceedings{Peck2013,
author = {Peck, E. M. and Jacob, R. J. K. and Chang, R.},
title = {{Using fNIRS brain sensing to evaluate information visualization interfaces}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {473--482},
url = {http://doi.acm.org/10.1145/2470654.2470723},
doi = {10.1145/2470654.2470723},
year = {2013}
}
@inproceedings{Harrison2013,
author = {Harrison, L. and Skau, D. and Franconeri, S. and Lu, A. and Chang, R.},
title = {{Influencing visual judgment through affective priming}},
booktitle = {Proc. ACM SIGCHI Conf. Human Factors in Computing Systems (CHI)},
pages = {2949--2958},
url = {http://doi.acm.org/10.1145/2470654.2481410},
doi = {10.1145/2470654.2481410},
year = {2013}
}
@article{Chuang2012,
author = {Chuang, J. and Manning, C. D. and Heer, J. },
doi = {10.1145/2362364.2362367},
journal = {ACM Trans. Computer-Human Interaction (TOCHI)},
number = {3},
pages = {1--29},
title = {{``Without the clutter of unimportant words'': Descriptive keyphrases for text visualization}},
url = {http://dl.acm.org/citation.cfm?doid=2362364.2362367},
volume = {19},
year = {2012}
}
}}}
See also [[R. Kosara's CHI blog post|https://eagereyes.org/blog/2015/conference-report-chi-2015]]
!Visualizing Data
!!~MatrixWave: Visual Comparison of Event Sequence Data (Paper, Honourable Mention)
Jian Zhao, Zhicheng Liu, Mira Dontcheva, Aaron Hertzmann, Alan Wilson
*weblog traces, paths through websites, attrition
*matrix encoding like Bertifier
*alternative to Sankey diagram
*explicit encoding using colour to encode differences in two matrix waves
*live demo
*highlight filtering
*multiple design alternatives for colour and explicit encoding
*user study w/ N=12 comparing matrix wave to Sankey
*q: Tim Dwyer: genetic quilts comparison from ~InfoVis (c. 5 years ago)
*q: applicable to other non temporal multi variate data? Like parallel sets?
*q: developer requirements
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702123.2702419]]
!!The Effects of Representation and Juxtaposition on Graphical Perception of Matrix Visualization (Paper)
Xiaotong Liu, ~Han-Wei Shen
*alternatives of matrix juxtaposition such as small multiples, matrix cubes
*comparing square and triangle matrix representation on tasks, overview, explore, confirm (latter two from previous work); no differences between square and matrix
*experiment 2: back to back juxtaposition relies on human symmetry detection
*experiment 2: complementary juxtaposition creates an asymmetric square matrix from two triangular matrices
*introducing new tasks in experiment 2
*experiment results overwhelming, hard to keep tasks in mind on results slide
*complementary matrix only useful for compact designs?
*basketball scenario use case: what are the links? Nodes?
*informal user study suggests designs require a loft of training
*no exploration of colour or interaction
*Q: J. Zhao: matrices scare users Interaction is key. How to mitigate? Give an overview, a starting point. No zooming in.
*Q: L. Harrison: can symmetry backfire?
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702217]]
!!g-Miner: Interactive Visual Group Mining on Multivariate Graphs (Paper)
Nan Cao, ~Yu-Ru Lin, Liangyue Li, Hanghang Tong
*finding groups that match multiple attribute values
*enterprise social network data
*hierarchical view like icicle plot, pie chart nodes in node link diagram
*"intuitive"
*hierarchical clustering in graph
*suddenly some matrix visualizations. Bar charts and adjacency matrices. A heat map design as well.
*disjointed designs. Not clear which tasks correspond to which designs.
*desert fog problem?
*node attribute editing dynamically changes graph topology. What is happening?
*ppt crashed.
*domain experts evaluation
*[[project page|http://team-net-work.org/]]
!!Trajectory Bundling for Animated Transitions (Paper)
Fan Du, Nan Cao, Jian Zhao, ~Yu-Ru Lin
*visual tracking tasks get hard with many targets
*three control points to bundle animations along midpoints of original positions, midpoints of paths, and midpoints of final positions
*pilot study with multiple animation factors: number of targets, dispersion metric, occlusions, trajectory lengths, deformation metric of target group
*multiple choice question in experiment (random guessing?)
*192 tasks comparing straight and bundled transitions (laughs)
*apparently more objects can be tracked with bundled trajectories, higher deformation is easier to track. High dispersion condition is more difficult with bundled trajectories. Simple tasks (what are these) are harder with bundled trajectories?
*q: R. Kosara: only targets are bundled, isn't this unfair? Colour could be used instead? Colour might be reserved for over dimensions is data?
*q: dynamic control tasks?
*q: NHR: what was accuracy
*[[paper|http://www.cs.umd.edu/~fan/papers/du2015chi.pdf]]; [[video|http://www.cs.umd.edu/~fan/videos/du2015chi.mp4]]
!Understanding & Evaluating Performance
!!~ModelTracker: Redesigning Performance Analysis Tools for Machine Learning (Paper)
Saleema Amershi, Max Chickering , Steven M. Drucker, Bongshin Lee, Patrice Simard, Jina Suh
*single interactive visualization as alternative to confusion matrices and precision-recall curves
*visualization updates when features added / removed, parameters tweaked
*item array unit visualization, interactive selection of elements and correcting of classifier decision
*FW: extending to multi-class environments
*[[project page|http://research.microsoft.com/apps/pubs/default.aspx?id=241307]]
!!How Good is 85%? A Survey Tool to Connect Classifier Evaluation to Acceptability of Accuracy (Paper)
Matthew Kay, Shwetak N Patel, Julie A Kientz
*use precision and recall rather than single metric of accuracy, weighted accuracy metrics highly dependent on context, application
*weather forecasting, home entry detection and text messaging / police contact
*[[paper|http://dub.washington.edu/djangosite/media/papers/Kay-AccuracyAcceptability-CHI2015.pdf]]
!Managing Personal Privacy
!!Privacy Tipping Points in Smartphones Privacy Preferences (Paper)
Fuming Shih, Ilaria Liccardi, Daniel Weitzner
*users less likely to divulge private information (location, context) when reason is given for private information request
!!~VeilMe: An Interactive Visualization Tool for Privacy Configuration of Using Personality Traits (Paper)
Yang Wang, Liang Gou, Anbang Xu, Michelle X Zhou, Huahai Yang, Hernan Badenes
*bar chart and uncertainty visualization for setting privacy controls for different circles or social media groups according to personality traits
*q: do personality traits measured from social media output correspond to actual personality traits?
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702123.2702293]]; [[video|https://youtu.be/FzuKiP4B1Tg]]
!I Like What I See - Interface Aesthetics
!!Computation of Interface Aesthetics (Paper, Honourable Mention)
Aliaksei Miniukovich, Antonella De Angeli
*dual speaker
*first impression: immediate reaction in .5s, deliberate first impression in a few seconds delayed impression ~4s
*screenshot based aesthetic computation, rather an element/code-based computation
*traditional aesthetics from fine arts vs. complexity based aesthetics from psychology (Reber et al 2004)
*8 metrics of complexity: visual clutter as proxy for set size, edge congestion, contrast (are colours wrong on screen? Can't see images), colour range, symmetry (vertical central symmetry), grid quality, white space
*study to tests these metrics with websites and apps, dozens of participants, hundreds of screenshots
*strongest metric was colour range, correlation-based analysis
*q: correlation b/w aesthetics and relative success of websites? Popularity of apps
*q: only one dependent binary variable? Only ugly/beautiful?
*q: Aaron Quigley: control for pre-attentive features? Modelling human vision rather than pixels on the screen?
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702575]]; [[video|https://youtu.be/Mt5ZO16FPpo]]
!!Patina Engraver: Visualizing Activity Logs as Patina in Fashionable Trackers (Paper, Best Paper Award)
~Moon-Hwan Lee, Seijin Cha, ~Tek-Jin Nam
*activity trackers
*piercing to engrave patina
*iterative engraving after radial use to produce personalized trackers based on calories burned, sleep, distance ran, etc.
*q: why the word patina?
*q: other wearables?
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702213]]; [[video|https://youtu.be/tYg_xJdAjls]]
!!Real-time Guidance Camera Interface to Enhance Photo Aesthetic Quality (Note)
Yan Xu, Joshua Ratcliff, James Scovell, Gheric Speiginer, Ronald Azuma
*real-time guidance for novice photographers on aesthetics of photos
*~MTurk study with novice and expert photographers
!!Infographic Aesthetics: Designing for the First Impression (Note)
Lane Harrison, Katharina Reinecke, Remco Chang
*Tufte quote from 1983 on infographic designers who use embellishments that have contempt for both information and audience
*is this contempt real?
*study involved showing infographics to participants to gauge consistency of aesthetic judgments: variable yet systematic
*what makes an infographic appealing?
*colourfulness matters, complexity matters when you split on gender (men don't care, women. Prefer less complex)
*example trial of 500ms exposure
*[[paper|http://reinecke.people.si.umich.edu/Publications_files/Harrison_CHI2015.pdf]]
!!ISOTYPE Visualization – Working Memory, Performance, and Engagement with Pictographs (Paper)
Steve Haroz, Robert Kosara, Steven Franconeri
*Austria museum following ~WW1, development of ISOTYPE, inventors wanted it to be a counting property, not size or area
*working memory limitations in counting: example of dot counting
*length judgment in bar chart is baseline, but focus is on counting problems: does showing quantity explicitly help? Yes, less error
*error difference between counting and bars closes when number of items is larger than working memory limits
*more accuracy with stacking rather an bars alone, icons have no effect
*1-back experiment to induce works memory load and delay
*icons benefit for long term memory
*stacking helps for fewer than 5 items, leverages working memory
*icons help long term memory
*icons are more engaging, people more inclined to look at ~ISOTYPE-style charts
*icons may hinder when not encoding data
*isotype not yet considered harmful
*[[project page|http://steveh.co/research/isotype]]
!Storytelling in ~InfoVis
*(filming session, did not take notes)
!!Storytelling in Information Visualizations: Does it Engage Users to Explore Data? (Paper)
Jeremy Boy, ~Jean-Daniel Fekete, Francoise Detienne
*A: No!
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702452]]; [[video|https://youtu.be/zDJwOdR8d3Q]]
!!Understanding Data Videos: Looking at Narrative Visualization through the Cinematography Lens (Paper)
Fereshteh Amini, Nathalie Henry Riche, Christophe Hurter, Bongshin Lee, Pourang Irani
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702431]]; [[video|https://youtu.be/ddEB5Ck_16A]]
!!How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques (Paper)
Anshul Vikram Pandey, Katharina Rall, Margaret L Satterthwaite, Oded Nov, Enrico Bertini
*[[paper|http://faculty.poly.edu/~onov/Pandey_et_al_CHI_2015.pdf]]
!!STRATOS: Using Visualization to Support Decisions in Strategic Software Release Planning (Paper)
Bon Adriel Aseniero, Tiffany Wun, David Ledo, Guenther Ruhe, Anthony Tang, Sheelagh Carpendale
*design study
*[[paper|http://hcitang.org/papers/2015-chi2015-stratos.pdf]]; [[video|https://youtu.be/qm57aHjTAYc]]
!Innovation in Theories & Products
!!From ~User-Centered to ~Adoption-Centered Design: A Case Study of an HCI Research Innovation Becoming a Product (Paper, Best Paper Award)
Parmit K Chilana, Andrew J Ko, Jacob Wobbrock
*the real "real world"
*what is the product?
*~LemonAid research prototype deployed during ~PhD, became ~AnswerDash
*adoption-centred design rather than user-centred
*integration of features of other tools, customization, analytics
*[[paper|https://faculty.washington.edu/wobbrock/pubs/chi-15.02.pdf]]
!Social Media and Mobile Camera Privacy
!!Somebody's watching me? Assessing e effectiveness of webcam indicator lights
Portnoff et al
*ratters: remote administration tools (~RATs), slaves, slave rental forums and malware repositories
*less than 50% of participants did not notice webcam indicator light turned on during study while using computer, not everyone understands webcam indicator lights: only 20% understand what indicator means, how many will act upon it?
*an overlay indicator that uses more screen real estate, animation (blinking), movement, and opacity results in a huge jump in participant's ability to notice the webcam
*indicators need to be noticeable from a distance when computer is open but not in use
!!From third place to surveilled place: the mobile in Irish pubs
Makoto et al
*mobile use as fact finder, mobile as truth (evidence for claim)
*loss of authentic banter, no one takes risks in debates, bets
*iPhone tower game
*the visible mobile seen as disrespectful
*the pub as surveilled place, photos and videos as evidence posted online
!Grip, Move & Tilt: Novel Interaction
!!Grip Change as an Information Side Channel for Mobile Touch Interaction (Note)
Matei Negulescu, Joanna ~McGrenere
*
!!An Experimental Comparison of Vertical and Horizontal Dynamic Peephole Navigation (Note)
Jens Müller, Roman Rädle, ~Hans-Christian Jetter, Harald Reiterer
*magic lens / peephole navigation with mobile/tablet
*vertical map for short term activities
*horizontal map for long term activities
!Sharing & Collaboration @ Work
!!~DocuViz: Visualizing Collaborative Writing (Paper)
Dakuo Wang, Judith S Olson, Jingwen Zhang, Trung Nguyen, Gary M Olson
*revision histories
*inspired by Viegas and Wattenberg's ~HistoryFlow, equal distance layout and timeline-scaled layout
*adopted by 100+ IP addresses, 200+ documents
*printed out visual results from actual use, qualitative coding of results printed on walls, several patterns emerge: outline, template, best-of-all, seismic activity
*[[paper page|http://docuviz.ics.uci.edu/]]; [[ACMl link|http://dl.acm.org/citation.cfm?id=2702517]]; [[video|https://youtu.be/TwXm9oS4CgY]]
!Visualizing Statistics & Graphs (Paper)
!!(s|qu)eries: Visual Regular Expressions for Querying and Exploring Event Sequences (Paper)
Emanuel Zgraggen, Steven M Drucker, Danyel Fisher, Robert ~DeLine
*medical data, sports data, weblog data / telemetry ecommerce
*aggregating events together to get counts, weblog event data
*canvas type environment, adding events to query types of event sequences, bar charts, heat maps, choropleth maps
*inserting wildcard nodes to queries, create or statements, dragging actions and queries from nodes to append to existing queries
*matching conditionals / back references
*integrating query systems with visualizations.
*BYOD study at MS employees
*[[project page|http://research.microsoft.com/apps/pubs/default.aspx?id=244463]]
!!Statsplorer: Guiding Novices in Statistical Analysis (Paper)
Chat Wacharamanotham, Krishna Subramanian, Sarah Theres Völkel, Jan Borchers
*lots of HCI papers violate stat assumptions, don't report stats correctly, etc. also a problem in medicine
*~StatWing, a few other related applications dating back to 1990
*wurst-case scenario, comparing calories and sodium of sausages
*q: different reporting standards? handholding from novice to intermediate?
*[[project page|https://hci.rwth-aachen.de/statsplorer]]
!!Investigating the Direct Manipulation of Ranking Tables for Time Navigation (Note, Honourable Mention)
Romain Vuillemot, Charles Perin
*multiple rankings, reordering, ranking changes over time
*using line chart to interact with rankings
*labels and context are removed when bump charts shown
*soccer rankings, IEEE ~InfoVis submissions
*limited accordance a
*[[toolkit on github|https://github.com/romsson/dragit]]
*[[project page|http://romain.vuillemot.net/publications/chi15-direct-manipulation-ranking-tables/]]
!!Dynamic Opacity Optimization for Scatter Plots (Note, Honourable Mention)
Justin Matejka, Fraser Anderson, George Fitzmaurice
*distribution of points has impact on suitable opacity level
*MOUP: mean opacity of utilized pixels. Find the individual point opacity which generates a MOUP of ~40%
*MOUP technique matches user-selected opacity selection
*[[paper page|http://www.autodeskresearch.com/publications/overplotting]]
!!Evaluating How Level of Detail of Visual History Affects Process Memory (Paper)
Eric D Ragan, John R Goodall, Albert Tung
*text analysis use case, different levels of visual detail and analysis process history
*experiment with process recall, retrospective memory bias? Different levels of visual detail
*high level of detail: all content and annotations shown
*annotations but not content as moderate levels of detail.
*low level of detail: annotations without annotated content
*no detail: black screen
*immediate recall in session 1, delayed recall following a week
*comparing explanation and actual record of what participants did
*more detail helps recall, even a low amount of detail helps.
*[[paper|http://people.cs.vt.edu/~eragan12/papers/Ragan_CHI2015.pdf]]
!Natural User Interfaces for ~InfoVis
!!Opportunities and Challenges for Data Physicalization (Paper)
Yvonne Jansen, Pierre Dragicevic, Petra Isenberg, Jason Alexander, Abhijit Karnik Johan Kildal, Sriram Subramanian, Kasper Hornbæk
*Hans Rosling talks using boxes and other everyday objects
*Mesopotamians used clay objects as data physicalizations
*Watson and Crick physical DNA model
*walkable age pyramid,
*what physical variables can be used to encode data?
*limitations: the laws of physics (analogs to 3d scatterplots)
*interaction is limited, like physical sorting, yet navigation is easy.
*animation and dynamic data: string to indicate network traffic at PARC, shape displays
*[[project page|http://dataphys.org]]
*Q: large data? Interacting with shape-changing displays. People still want to touch
!!Exploring Interactions with Physically Dynamic Bar Charts (Paper)
Faisal Taher, John Hardy, Abhijit Karnik, Christian Weichel, Yvonne Jansen, Kasper Hornbæk, Jason Alexander
*EMERGE system, shape changing displays
*Heer and Shneiderman's taxonomy of tasks used to guide evaluation with 17 users
*[[paper|http://www.scc.lancs.ac.uk/~jason/pdf/CHI15_EMERGE.pdf]]
*hiding and comparing rows and columns of the bar chart
*dragging rows to new positions, exploring design space of interactions
*navigation through direct and indirect means, paging operations
*surprise factor, users not afraid of touching dynamic bars: physical based interactions good for annotation and filtering, gesture based interactions good for organization and navigation
*Q: collaboration? Comparison against 2D? During study, were questions asked of participants about the data?
!!Evaluating the Memorability of Physical Visualizations (Note)
Simon Stusak, Jeannette Schwarz, Andreas Butz
*RW on memorability of physical and haptic sensations
*compared 2d display on iPad vs. physical display of a grouped bar chart
*study: reading phase, immediate recall phase, delayed recall phase
*extreme value questions, numeric value questions, facts TF
*no significant differences overall, but statistically significant differences for extreme values
*participants spend a little more time when reading physical visualizations
*no haptic exploration
*[[ACM link|http://dl.acm.org/citation.cfm?doid=2702123.2702248]]
!!Personality as a Predictor of User Strategy: How Locus of Control Affects Search Strategies on Tree Visualizations (Note, Honourable Mention)
Alvitta Ottley, Huahai Yang, Remco Chang
*internal and external locus of controls differ in terms of their search strategies with dendrograms sand indented trees
*externals provide interface with implied search strategy
*LOC affects performance and strategy
*[[paper|http://goo.gl/N7dQqR]]
!!~SketchSliders: Sketching Widgets for Visual Exploration on Wall Displays (Paper, Honourable Mention)
Theophanis Tsandilas, Anastasia Bezerianos, Thibaut Jacob
*synchronized custom sliders with wall displays. Not sketching the Vis themselves, but the interactions
*branched sliders for focus and context within parent sliders
*transformation functions with sliders
*sliders as analytical provenance
*evaluation with 6 researchers, think aloud
*sketch yr own interactive slider controls on tablet to control wall-based #infovis #chi2015 [[paper|http://insitu.lri.fr/~fanis/docs/SketchSliders.pdf]]
*unable to change or refine sliders after drawn
!Interactive and ~Multi-Surface Maps
!!~TerraGuide: Design and Evaluation of a ~Multi-Surface Environment for Terrain Visibility Analysis (Paper)
Matthew Oskamp, Christophe Bortolaso, Robin Harrap, T.C. N Graham
*presented by Nick
*can landmarks be visible from points on a map? Terrain analysis for railroad construction, campus design, cell phone tower placement
*contour maps
*tightly coupled views between table, wall, tablet. On table, viewshed using colour encoding to represent visible terrain. Low resolution realtime viewshed computation, high resolution upon delay
*panaoramic view on wall monitor computed in real time
*helicopter view from handheld tablet, shadow of tablet projected onto table
*are combinations or views effective?
*viewshed and panoramic view performed worse than viewshed alone, but users more confident with both. Panoramic view attended to seldomly. Correspondence problem between viewshed orientation and orientation of user to panoramic view.
*tightly coupled view backfired.
*viewshed + tablet view considerably better results. Switching between tablet and viewshed visualization occurred often.
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702480]]; [[video|https://youtu.be/iKrPvpeWi_Y]]
!!Lightweight Relief Shearing for Enhanced Terrain Perception on Interactive Maps (Paper, Best Paper Award)
Wesley Willett, Bernhard Jenny, Tobias Isenberg, Pierre Dragicevic
*kinetic depth cues when dragging a map to enhance terrain perception
*most relief maps are totally custom, hand-drawn
*motion parallax, depth cues, shape cues easily extracted
*depth cues with perspective changes in Google Earth, but difficult to control theses views and camera placement. Zoom, translation, rotation, all difficult to control and distracting users from primary task
*interactive relief shearing
*plan oblique relief
*explicit drag shearing in direction of drag + snapping back into position, animation cues, spring model
*coupling shearing to perspective of device in a tablet display
*experiments: hiking trip in the Alps. Task, given two points on a map, which is higher.
*participants more accurate with animated and integrated shearing than with static maps
*experiment 2: like before, but adding panning. Interested shearing prevails, more time required. Speed accuracy tradeoff
*ambient shearing may be useful to support large displays, many viewers
*use in the wild may vary in performance due to differences in lighting direction
*FW: flexible displays?
*q: earthquake simulation? Time in experiments?
*[[paper page|http://aviz.fr/reliefshearing]]
*device tilt a different activity
!!An Evaluation of Interactive Map Comparison Techniques (Paper, Honourable Mention)
~María-Jesús Lobo, Emmanuel Pietriga, Caroline Appert
*comparing road maps vs. topographic maps, satellite images, etc.
*4 existing conditions: juxtaposition, translucent overlay superposition upon explicit reveal, blending lens, swipe reveal
*offset lens: juxtaposed lens novel technique
*vision driven vs. motor driven scanning
*divided attention vs. visual interference
*superposition more efficient than juxtaposition
*vision driven strategy more efficient
*offset lens did not perform as well as hoped
*translucent overlay and magic lens techniques perform best
*[[paper page|http://ilda.saclay.inria.fr/mapmuxing/chi2015/]]
!!Ethermap - Real-time Collaborative Map Editing (Paper)
Thore Fechner, Dennis Wilhelm, Christian Kray
*non-blocking real-time collaborative editing of geospatial data
*shape geometry editing can be blocking
*requirements: user awareness, history, communication, and realtime synchronization
*all geometry edits propagated to other users
*user study and expert interviews
*role-based interaction noticed in evaluation.
*FW: what is effect of group size?
*[[ACM link|http://dl.acm.org/citation.cfm?id=2702536]]
*[[online demo|http://ethermap.uni-muenster.de]] map name: ~CHI_2015, username: Your Username
!Interacting with ~GUIs
!!Color Portraits : From Color Picking to Interacting with Color (Paper, Honourable Mention)
Ghita Jalal, Nolwenn Maudet, Wendy E Mackay
*interviewing visual artists
*rethinking colour palette design
*retaining dimensions (lightness,saturation) of a palette while varying another (hue)
*palette histories
!Paper talks I wanted to see but didn't
!!VIZMO Game Browser: Accessing Video Games by Visual Style and Mood (Note)
Jin Ha Lee, Sungsoo (Ray) Hong, Hyerim Cho, ~Yea-Seul Kim
*
!!Stock Lamp: An ~Engagement-Versatile Visualization Design (Paper)
Yuzuru Tanahashi, ~Kwan-Liu Ma
*
!!Your Paper is Dead! Bringing Life to Research Articles with Animated Figures (altchi)
Tovi Grossman, Fanny Chevalier, Rubaiat Habib Kazi
*
!!~BodyVis: A New Approach to Body Learning Through Wearable Sensing and Visualization (Paper)
Leyla Norooz, Matthew Louis Mauriello, Anita Jorgensen, Brenna ~McNally, Jon E Froehlich
*
!!Supporting Subtlety with Deceptive Devices and Illusory Interactions (Paper)
Fraser Anderson, Tovi Grossman, Daniel Wigdor, George Fitzmaurice
*(magician)
!!Apparition: Crowdsourced User Interfaces That Come To Life As You Sketch Them (Paper)
Walter S Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P Bigham, Michael S Bernstein
*
!!“Everyone Is Talking about It!”: A Distributed Approach to Urban Voting Technology and Visualisations (Paper)
Lisa Koeman, Vaiva Kalnikaitė, Yvonne Rogers
*
!!Break It Down: A Comparison of Macro- and Microtasks (Note)
Justin Cheng, Jaime Teevan, Shamsi T Iqbal, Michael S Bernstein
*
!!Designing for Citizen Data Analysis: A ~Cross-Sectional Case Study of a ~Multi-Domain Citizen Science Platform (Paper)
Ramine Tinati, Max Van Kleek, Elena Simperl, Markus ~Luczak-Rösch, Robert Simpson, Nigel Shadbolt
*
!Links
*[[CPSC 508 homepage|http://www.cs.ubc.ca/~feeley/cs508/index.html]]: taught by [[Mike Feeley|http://www.cs.ubc.ca/~feeley/]]
*[[CPSC 508: Piazza|https://piazza.com/class#winterterm12012/cpsc508/20]]
<<list filter [tag[cpsc508]]>>
!Reviews
!!<<cite Dennis1966 bibliography:Bibliography-CPSC508>>: Multiprogramming semantics
Dennis and Van Horn's article describes a set of operations to be included in the design of supervisor programs for multiprogrammed computer systems. These operations, referred to as meta-instructions, permit flexible and efficient operation of multiprogrammed computer systems with respect to their defining characteristics: parallel computations, multiple users, sharing of hardware resources and data, as well as the protection of processes and saved data from other processes. The meta-instructions dictate an etiquette, a formal communication between users and processes, without making explicit references to the shared hardware resources.
The 1983 preface was helpful for illustrating the importance of the ideas presented in the article, and for defining some of its vocabulary. Nevertheless, I have a difficult time imagining the impact that this article made at the time it was published, 46 years ago. This is in part due to the structure of modern computer science undergraduate curricula: some schools adopt an approach that mirrors the historical development of computing systems and programming languages, while others begin with higher-level languages and abstractions, without referring in depth to the historical justification for their design. Because I was a student of the latter approach, I cannot easily envision what came before this article, the alternative ideas of the era, and in which regards were Dennis and Van Horn's ideas improvements. My inability to gauge the impact that this article made in 1966 is also due in part to the descriptive tone of the paper; it does not assume an evaluative or comparative stance against alternative sets of meta-instructions or competing designs of supervisor programs.
The authors do not claim to present a complete set of meta-instructions, indicating that "additional operations will prove necessary for a specific [multiprogrammed computer systems]" (p.29). Are we to infer from this that these meta-instructions are complete for a general description of a MCS? I was anticipating a section toward the end of the article that would indicate areas for future work, such as additional or alternative meta-instructions yet to be defined and evaluated. Instead, the article ends with a short example of directory access.
Finally, I'll admit to be confused and overwhelmed while reading the description of protected entry points, where it became difficult to keep track of multiple processes, procedures, routines, computations, and entries; fig. 5 didn't help in this regard. My understanding of this section is that the use of these particular meta-instructions allow a new process to be granted capabilities not shared by its parent process.
!!!!Discussion Questions
*How did the multiprogrammed computer systems discussed in the introduction [R1,R2,R3] handle protection, parallelism, and multiple principals?
*The article described segments as the smallest unit on which a capability can be granted. Today, individual files are the smallest unit on which a permission can be granted. The authors did not explicitly commit to a 1:1 correspondence between segments and files. If a file could be composed of multiple segments, what are the implications of having different sets of capabilities on different parts of the file?
*How have the terms "process" and "computation" changed meaning since 1966?
*Why are airline agents not considered to be principals (p.30)? If they can access (read) and perform operations on a centrally stored data base (write), does this not make them a principal with read/write capabilities?
*In 1966, did there exist alternative approaches to directory structure? How did they differ from a hierarchical structure?
*If principal L removes a segment still in use by principals A, B, and C (p.34), how is ownership of that segment assigned?
!!<<cite Hansen1970>>: System nucleus
Hansen's short article describes how a hierarchy of processes, both internal and external to a multiprogramming system, communicate with one another. The structure of the article is such that it begins at the low level of individual processes and ends at the high level of a process within an operating system, which is in turn defined by a set of rules contained by a system nucleus. At this point the suggestion of a static rule set (the nucleus) and a replaceable operating system becomes possible, this being the key contribution of the article.
When discussing multiple processes, the author places emphasis on communication and cooperation, rather than making explicit references to sharing, withholding, and protection, the roles of the meta-instructions in Dennis and Van Horn (1966)'s article. There is is one reference to protection in the implementation section (p.250), wherein excessive times relating to internal processes are due to the RC 4000's "peculiar storage protection system". I'd like to know how this system is peculiar, and how did it differs from the protection mechanisms described by <<cite Dennis1966>>?
Like the <<cite Dennis1966>> article, this article adopts a largely descriptive, rather than comparative tone. The process communication as described is more nuanced than binary lock/unlock semaphores [R3]. Aside from this, I cannot infer comparisons on other levels, such as how the RC 4000's hierarchically-organized operating system and nucleus differs from the organization of other multiprogramming systems of its time. Finally, the implementation details do not include a comparison against previous benchmarks or alternative systems, so it is difficult to gauge the significance (in 1970) of the gains made by this implementation.
!!!!Discussion Questions
*How were comparative evaluations conducted between operating systems in 1970?
*The article does not make mention of segments. By 1970, were these taken to mean the same as files? as data?
*Why is the storage protection system of the RC 4000 considered to be "peculiar"?
*In Section 4 (Process Communication), column 2, paragraph 2, I'm seeking a clarification of what is meant by "the system [udes two primitives that enable a process to wait for the arrival of the next message of answer and serve its queue in any order". What are the two primitives that the author is referring to?
!!<<cite Daley1968>>: Multics
Daley and Dennis make a number of contributions in this paper. With MULTICS, they introduced several concepts that are retained by modern operating systems: virtual memory, dynamic linking, and external symbolic references to procedure and data. Together, these concepts allowed for code reusability and the sharing of stored data, resolving inefficiencies inherent to the use of early multi-program, multi-user computers. It was no longer necessary to make copies of shared procedures and data. Nor was it necessary to know the location of shared procedures and data in the storage hierarchy, as these could be referenced by symbolic names and evaluated at runtime. In this system, procedure segments could not be self-modifying; they could be shared by multiple processes and be invariant to the recompilation of other segments.
The main body of the article provides technical details of how these concepts are realized in MULTICS. Unlike <<cite Dennis1966>>, this article goes beyond high-level concepts and multi-programming semantics to discuss how procedures are carried out with regards to processor design and machine-language instructions. The authors outline how generalized addresses for instruction fetches and data are formed given different instruction formats, and how remote procedures are called and returned from using the stack.
While there are a small number of references to other competing concepts of the era, such as overlay techniques, the article does not adopt a comparative tone. I had a similar impression of <<cite Dennis1966>> and <<cite Hansen1970>>; I could acknowledge the major contributions and the concepts that have shaped modern operating systems, but I'm left wondering what came before. I'm curious to know how each of their design decisions in MULTICS responded to a problem in the operation of other multiprogramming systems, their processor design, instruction format, and procedure call/return routine.
!!!!Discussion Questions
*On p. 307 the authors state that it is unnecessary to address information via files, and that no distinction be made between files and segments. The MAC system, described in <<cite Dennis1966>>, presumably addressed information via files, which they describe as long-lived segments. Was the question of segment duration no longer relevant by the time of the current article (1968)?
*I'd like to know more about procedure overlays, complicated buffering, and the and the direct use of associative hardware, each alluded to as being used in other systems; how they differ from the mechanisms of MULTICS?
*Terminology: are routines and procedures interchangeable terms? Is a routine a chain of procedures?
!!<<cite Saltzer1974>>: Multics follow-up
Here we have a follow-up paper to <<cite Daley1968>>'s Multics paper, one that is markedly different in tone and emphasis. Refreshingly less technical than <<cite Daley1968>>, Saltzer's article gives a precedence to the human factors relating to the organizational adoption of a Multics system, which I appreciated, being a human factors researcher. A recurring theme in this article is the question of trust: how much of it should be placed in the system administrator, in individual users, and in their supervisors?
The article begins in a similar fashion to that of the previous papers read in this course, with a description of design principles that have guided the development of Mutlics. I found it interesting that each principle reflected the organizational structure as well as the expectations and privileges of individual workers. Access to information was to be maintained on a need-to-know "least privilege" basis, protection mechanisms were based on "permission rather than exception", and protection specifications were decentralized. While reading this paper, I could not ignore the historical context of the paper; references to military and government protocol allude to Cold war era security policies and fears that centralized system administrators could become compromised.
The following sections explain the logistics of hierarchical naming as well as the flexible specificity of access control for stored objects. The author touches upon the question of who "owns" an object or segment, a topic that is also discussed by <<cite Wulf1974>>; should a subordinate own objects that they create? There is also a section on authentication and login auditing, which considers how a user may interact via a terminal, access protected subsystems with further authentication, or login anonymously.
I found the section on primary memory protection to be less interesting than the preceding sections, which is expected given my interest in human factors. The key concepts discussed in this section were that of flexible and extensible protection mechanisms, descriptor segments, protected subsystems, and rings of protection, the latter recalling <<cite Dennis1966>>'s "spheres of protection". I found the writing in this section to be less clear, likely due to the inherent complexity of descriptor segments. An example: "it is worth nothing that even the writing of the descriptor segment is done with the use of a descriptor for the descriptor segment itself" (p.398).
I appreciated the generous discussion of design alternatives and tradeoffs throughout Saltzer's article, an element that I felt was lacking in previous articles read in this course. The author compares Multics with an earlier system, CTSS, discussing the differences between links/capabilities and access control lists. He also discusses the pros and cons of permitting trap procedures, ultimately deciding that an excess lack of trust could be just as crippling as placing too much trust in individual users. He compares hierarchical control of access control lists to self-control policies and private work areas, the latter necessitating a system "locksmith". Wisely, he admits that this design decision is a tradeoff that should reflect the policies of the organization owning the installation.
Finally, I enjoyed reading about the limitations and potential weaknesses of Multics. This section is honest in its consideration of these weaknesses, their expected causes, and their foreseeable solutions. The author cleverly states that the mechanisms of Multics are not where weaknesses reside; they are rather in the misinterpretation or misimplementation of these mechanisms; mismatches between system mechanisms and organization policies. He returns to this in the conclusion, stating that protection mechanisms make a system securable, but not inherently secure; that is the task of the organization. The other topic I paid special attention to in this section was that of the complexity and inadequacy of the user interface. I considered that Saltzer's article was published 8 years before the first ACM conference in human factors (SIGCHI), a venue where the research community could address issues of user interaction head-on, with particular attention directed toward mismatches between system models and users' mental models (p.399). The latter issue is also quite relevant to the discussion of HYDRA (<<cite Wulf1974>>), which gave rise to situations where a user's mental model could be markedly different from the underlying system model.
!!!!Discussion Questions
*How were backpointers (p. 396) implemented? How did the propagation of changes to access control lists work?
*I was somewhat confused by the section relating to "locking up" the supervisor and "source and sink" input/output operation. Can anyone clarify what idea(s) I'm supposed to retain from this section?
*Re: weaknesses and limitations (p. 400) - can the role of a system operator really be seen as a utility? Isn't this type of centralized bottleneck the designers were striving to avoid?
!!<<cite Wulf1974>>: Hydra
Wulf and colleagues at Carnegie Mellon present HYDRA, a kernel, elsewhere referred to as a system nucleus (<<cite Hansen1970>>), for a multiprocessor operating system. The objective of the kernel is to allow for simultaneous existence of multiple concurrent operating systems, built from common mechanisms but differing in the policies that dictate how the mechanisms are used. As an analogy, the kernel provides a set of Lego building blocks but not the instructions for their arrangement in a system model; some pairs of blocks will have corresponding interlocking parts, while others do not. They posit that these mechanisms should include facilities for creating, sharing, protecting, and interacting with typed objects; these are policies, not in the jurisdiction of kernel mechanisms. Recalling <<cite Saltzer1974>>, "the structure of extant operating systems bears a remarkable resemblance to that of the organization which created them", dictating a need for the separation of mechanism from policy, while providing flexibility and extensibility. I agree with their view that "protection is a mechanism; security is a policy" (p. 340).
Like <<cite Saltzer1974>>, this article touches upon human factors issues, namely that of the users' mental models. Here the mental model, being of directories and objects, does not match the system model, having no hierarchical structure. Furthermore, one user's mental model of the system may differ substantially from another user's mental model. It may very well differ for a single user depending on their mode of use, their current choice of operating system. This flexibility has a drawback; if users cannot share a consistent mental model of the system, communication and collaboration between users breaks down over time. Hierarchies and ownership are concepts that are easy to understand and thereby can be more easily communicated.
While less comparative and reflective in tone than <<cite Saltzer1974>>, the authors do draw comparisons with similar capability-based systems, namely <<cite Dennis1966>>. These comparisons are helpful in explaining how procedures are called and returned from.
What I enjoyed most about this paper was the example. It served to clarify the relationship between capabilities, type-dependent procedures, local name spaces, and typed objects. However, I'm curious if this use case came out of a need of the authors' organization or if it was entirely speculative. Either way, it seems to be a plausible scenario.
!!!!Discussion Points
*"Objects", as discussed in this article, are markedly different from the notion of objects discussed by <<cite Saltzer1974>>. The concepts of objects and types are flexible, bringing to mind object-oriented programming and polymorphism. This flexibility would become the jurisdiction of application software, and not that of the operating system. Consider the bibliography example: today, software would be responsible for the creation of such typed objects and for providing the affordance of type-specific procedures.
*I became somewhat confused by the discussion of local name spaces, namely the paragraph on p.339 which contains "the [local name space] for a particular invocation is the result of combining the caller-independent capabilities (listed in the procedure object) with caller-dependent actual parameters (only characterized in the procedure object)"; perhaps it is a subtle distinction to make, but I'm curious to know why the former is "listed" while the latter is "characterized".
*The ultimate utility of the walk primitive is not yet clear to me. I was hoping that this would appear in the example. Can anyone provide an example of how this would be used in practice?
*The authors firmly reject the concept of object "ownership" and the hierarchical arrangement of objects. This is quite a firm stand and a different attitude from that of previous articles read in this course. I'm still not sold on why ownership and hierarchies are inherently evil, but I'm open to hearing interpretations from other members of the class.
!!<<cite Levin1975>>: Policy/mechanism separation in Hydra
Levin and colleagues' follow-up paper to <<cite Wulf1974>>'s Hydra article explores the design philosophy of policy-mechanism separation in detail. While <<cite Wulf1974>> described the Hydra kernel in a broad sense, providing an overview of the protection mechanism, systems and subsystems, objects and types, culminating in an extensive usage example, the focus of <<cite Levin1975>> is that of the policy-mechanism separation in particular.
The policy-mechanism separation is evident in three aspects of Hydra: disk scheduling, paging, and protection. In these regards, the kernel should provide the facilities for various operating systems to establish their own scheduling, paging, and protection policies, thereby allowing multiple concurrent modes of operation in a multiprocessing system. Their aim was to reduce the number of facilities provided by the kernel to a successful minimal set; recall that this was also an objective of the MULTICS system designers (<<cite Saltzer1974>>), who struggled to reduce the number of program modules in the most protected area. After returning to this paper upon reading <<cite Cao1996>>, we now see that policy decisions need not be restricted to the choice of either the kernel or the operating system(s); it is possible to leave some policy decisions, namely those involving disk scheduling and memory management, to application processes themselves.
I return now to the contents of <<cite Levin1975>>'s article, which delves into great detail regarding how policies are enacted at the user level in the Kernel Multiprocessing System (KMPS). Much of how this occurs hinges upon policy modules and the processes they manage. Processes themselves cannot dictate scheduling decisions, as the overhead involved with switching protection domains would be substantial. Instead, these decisions are negotiated by policy modules, which serve as meta-processes that can perform several kernel-defined operations relating to the starting, stopping, and restarting of processes. The authors first describe how this is accomplished with a single policy module, and then with multiple policy modules, each associated with a number of processes. Here they explain that "the kernel is not devoid of policy", as there must be an overarching policy to ensure fairness among multiple policy modules.
The policy-mechanism separation discussion continues in the context of paging or cache management, a topic covered in great detail in <<cite Cao1996>>. As large user programs require that pages be moved in and out of the cache, the kernel provides facilities to allow a policy module to set the paging policy for the processes under its control. Again there exists a kernel-level policy that maintains a guarantee of fairness between multiple policy modules.
The last section of the article deals with protection, where the authors state that "in contrast to scheduling and paging, the Hydra protection structure provides a clear separation of mechanism and policy". I find it amusing that in only one out of three Hydra design aspects discussed in the article do the authors claim that policy and mechanism is actually separable; I feel as though the title of the article should be "Struggling to separate policy and mechanism in Hydra". Despite their claim, when it comes to protection, the authors later state that the kernel still maintains the role of "a guaranteer of a negotiation"; it does not decide upon the policies involved in the negotiation, but enacts the meta-policy that ensures that a negotiation takes place. I feel that this captures the overall message of the article: there can be considerable flexibility at the user level, however the kernel cannot be entirely devoid of policy.
!!!!Discussion Points
*<<cite Levin1975>>'s article assumes an understanding of how semaphores work, and how a process blocks on a semaphore. I'd appreciate a simple clarifying explanation of what the authors are referring to here; how is this different or similar to the message-passing between processes described by <<cite Hansen1970>>?
*A similar assumption is made about an understanding of what a process' processor mask does. What is the origin of these and how do they operate in practice?
*It appears that while many decisions can be left to policy modules, they cannot be trusted with a page replacement policy, a problem that is more complex than the authors initially let on (p. 138). Here is an instance of where the authors, like many of their contemporaries, struggled with reducing the number of facilities offered by the kernel to a minimal set.
!!<<cite Cao1996>>: ACFS / ~LRU-SP
We have leapt forward in time; more than 20 years after <<cite Levin1975>>'s article, Cao and colleagues address some of the same issues: policies of memory management and disk scheduling, and what part of the system is ultimately responsible for defining and enacting these policies. In this article, Cao and colleagues describe the Application-Controlled File System (ACFS). While <<cite Levin1975>> recounted the intent to offload the duties of policy-making and policy-enacting to policy modules, Cao and colleagues go a step further with ACFS, allowing application processes to govern memory management and disk scheduling policies themselves.
I applaud the structure of this paper, as it begins with a discussion of how application-controlled policies are realized in the context of a single process, followed by the case of multiple processes. Along the way, we learn the rules that form the basis of these policies (section 2.12), and we can compare application-controlled prefetching and caching with common memory management policies found in other systems, namely the global least-recently-used policy. I found the diagrammatic figures helpful for understanding the controlled-aggressive policy, as well as how processes can flexibly control their own memory management policies, replace blocks in memory, and assign placeholder blocks. I also appreciate that the authors accounted for oblivious processes (those using the kernel's default policy), foolish processes (those using a worse-than-default policy), and smart process (those using a better-than-default policy); even in the case of foolish processes that mismanages its own resources, the "do no harm" rule is not violated, and other processes are not affected.
I felt somewhat bogged down in the section on implementation details, that the pacing of the article slowed and I began to lose sight of the higher-level conceptual understanding of ACFS. This section could have been shortened, with low-level implementation details provided in an appendix or material supplemental to the article.
I was looking forward to reading about the empirical results given in section 5, which compares ACFS performance against other system designs for both single and multiple processes. Admittedly, I was disappointed in that I could not gauge the significance of their results. I am accustomed to reading about empirical studies in the human factors literature, where if multiple results are stated to be significantly different from one another, statistical evidence is provided in the form of confidence intervals, p values, and measures of effect sizes. In this article, I am told that ACFS is faster and requires less block I/Os than competing system designs, however I am unsure as to whether these differences are reliable and practically significant. The authors mention that the results were averaged over several runs, and in some multi-process cases the variance was quite substantial (they admit to not knowing why this is so, p. 338); it would be informative to see confidence intervals superimposed over the bars in figures 4-11. I'd also be curious as to whether the results of several runs were normally distributed; it is possible that the mean elapsed time and number of block I/Os are not reliable metrics.
The paper concludes with a related work section; the placement of which makes sense given the need to first explain the workings and performance of ACFS. I was frustrated that previous papers read in this course did not have such a section, making it hard to gauge which aspects of systems being discussed formed novel contribution(s) to the field. Of course, it's likely that between the mid-1960s and mid-1970s, there was little related work to cite or compare against.
!!!!Discussion Points
*Is it common for empirical studies in systems papers to be presented in this manner? Is it sufficient to make claims of significant gains in performance as the authors have done have, asking readers to take them at their word?
*Some questions are raised in the final paragraphs of the conclusion (p.341). The question of non-uniform fetch time caught my eye. What are the implications for ACFS if fetch times are not consistent? Didn't they fetch from "a real disk" in their study?
*I'm confused by the final paragraph; the questions raised here about "performance with real-life workloads" and the interaction between "prefetching, cache management, and CPU scheduling" - were both of these questions not already addressed in the current study?
!!<<cite Saltzer1984>>: End-to-end argument
Ten years after Saltzer's Multics article, <<cite Saltzer1984>> wrote this short position paper that channels a line of thinking now familiar to us, one that questions low-level function implementation. <<cite Levin1975>> advocated the separation of mechanism and policy: the former is to be provided at the lowest level by the kernel, while the latter is the responsibility of higher-level policy modules. <<cite Cao1996>> went further, demonstrating that applications could be left in control of policies relating to scheduling and memory management. <<cite Saltzer1984>>'s argument fits nicely within this discussion by asking the question: what value would implementing a function at the lowest level bring? Could it alternatively be completely and correctly implemented at a higher-level application without significant performance costs?
I've just recounted Saltzer's "end-to-end" argument, one that "has been used for many years with neither explicit recognition nor much conviction". The article's contribution is this recognition, illustrated by several examples in which the argument applies: in communication protocols, encryption, duplicate message suppression, system crash recovery, real-time conversation, and message delivery acknowledgement. In the examples that follow, it becomes clear that the argument gives way to a simple design principle: a function should only be implemented at the lowest level for the purpose of performance enhancement, and not to ensure correctness and perfect reliability, which is the duty of the application. This is a trade-off, and the authors warn that low-level function placement may not always result in performance gains; applications that do not need the function will "pay for it anyway", other applications will require a tweaked custom version of the function, and low-level subsystems may have incomplete knowledge of what is occurring at the application levels, resulting in a loss of efficiency. There is a question of weighing implementation cost and a need for absolute correctness. If a system is required to be entirely devoid of errors, as in cases where human lives depend on system correctness, an expensive lower-level function implementation may be necessary. However, if rare errors can be tolerated within some threshold, such as in the case of real-time speech communication, a costly low-level implementation is overkill; applications should perform their own error-checking and recovery.
<<cite Saltzer1984>> draws a comparison between the end-to-end argument and the case for a reduced instruction set computer (RISC) architecture. Both question the necessity of providing a large number of specific functions at a low level. Providing a minimal yet generalizable set of functions gives way to a system that is more flexible for applications.
I'll also point out that I found this article to be particularly well-written. The argument and related design principle were both clearly stated and grounded in several convincing examples. I also found the placement of the "History, and application to other system areas" section to be effective for indicating the generalizability of this argument.
!!!!Discussion Points/Questions
*There's just a fleeting reference to Multics in this article (in the History section, p.8). By 1984, was Multics already abandoned as a research project?
*Regarding protection functions, should applications maintain their own access control lists? Should the question of file/segment ownership be left to the application, deciding whether ownership is necessary and if so, who gets it?
!!<<cite Clark1985>>: Upcalls
This 1985 article by MIT's David Clark pertains largely to inter-process communication. It presents an alternative to subroutine calls and asynchronous communication between processes; this alternative is one of synchronous procedure calls between layers of a system. This design choice reflects the observation that the flow of control of a system often proceeds in both bottom-up and top-down directions, that lower-level subsystems must occasionally have awareness of application-level activity, and vice-versa. It also reflects the need for parallel execution of tasks. This design decision spreads the responsibility of system and inter-process integrity more evenly across levels of abstraction. The discussion of placing more control at application levels recalls the end-to-end argument of <<cite Saltzer1984>>. While Saltzer's article took the form of a position statement, Clark addresses the issue from a practical standpoint: he considers how processes and layers should communicate with one another, as well as a programming methodology for implementing these communication channels. He proceeds to demonstrate the use of this methodology in a system called Swift, an initiative motivated by earlier research relating to the implementation of network protocols. The article belabours the efficacy of his programming methodology in this application area; however the author later admits that this methodology may not be appropriate elsewhere.
I had a difficult time with this article. Above all, it lacked clarity of exposition. For instance, I often look forward to reading longer case-study examples when used to explain abstractions, such as Clark's programming methodology. Clark missed the mark with his network protocol example; it did not effectively illuminate the methodology. I found myself annotating the code fragments in Fig. 2 and 3 myself, tracing the upcalls and downcalls that he described in the body text on the preceding page. Clark should have done this work for the reader: he could have provided more informative figure captions and in-line code comments, as well as annotations indicating the flow of execution. The structure of the example was also frustrating: I would had an easier time understanding the example had it been prefaced with Figure 4, albeit with a more informative caption, used as a guiding overview; this figure was more helpful than the code examples. Given that the author wasn't constrained for space (whitespace on p.180, the superfluous Fig. 1), the example could have also benefited from a side-by-side comparison with an inappropriate former methodology involving asynchronous communication and subroutine calls. Alternatively, he could have used this opportunity to indicate how his programming methodology could be abused by an inexperienced or sloppy programmer, to compare the right and wrong way to use his methodology: what would the parallel programming equivalent of spaghetti code look like for this example? Despite these problems, I'm at least grateful that the example appeared early in the article, instead of being tacked-on at the end for confused yet patient readers, which was the case of the bibliography example in <<cite Wulf1974>>'s article. In addition to the subject matter of the example, I found that the author assumed a considerable working knowledge of network protocols throughout the article, which further contributed to the overall lack of clarity. Finally, I felt that section on related system features (থlt rushed and tacked-on, describing important system-wide issues relating to task scheduling and address space management. These aspects deserved to be treated in greater detail, perhaps with additional examples of the programming methodologies with comparisons to former approaches.
The title of this article by Clark is somewhat of a misnomer. The paper's focus was a programming methodology for network protocol subsystem. The methodology relies just as much on downcalls and inter-process communication via multitask modules as it does on upcalls. However, the suggestion of upcalls is provocative, perhaps a "substantial heresy" to some thinkers. The title's "structuring of systems" also lays claim to a much wider scope than to which Clark's methodology applies. Doubtless this article's title and motivating philosophy was controversial; and yet the body of the article turned out to be fairly humble. Clark later outlines the limitations and pitfalls of the methodology, and concludes by stating that the scope of application areas where this methodology may be appropriate is likely to be somewhat restricted.
!!!!Discussion Points/Questions
*The conclusion suggests that different programming methodologies may coexist in a modular system. The Swift methodology could be used for highly parallel subsystems, while other methodologies would exist elsewhere. What happens when tasks from parallel subsystems communicate with other system areas? What is the mediating methodology?
*What does Clark mean by " asking advice" from a higher-level layer?
*Can someone show me alternatives to the code examples in Fig 2,3? I'd like to see (a) the former way of implementing such a protocol; and (b) the sloppy/abusive adoption of Clark's upcall methodology.
!!<<cite Redell1980>>: Pilot
Pilot is an operating system for a personal computer, developed by Redell and colleagues at Xerox in the late 1970s. This article describes the operating system in great detail, distinguishing it from the design of operating systems for multi-user installations. A critical aspect of the Pilot operating system is that it is written entirely in the Mesa programming language. Mesa also depends on Pilot for runtime support, so the two are mutually interdependent. The the main body of the article is structured along the separation of Pilot's //interfaces// and their //implementation//. The former is specified by definition modules, while the latter is specified by program modules in the form of algorithms and data structures; an implementor program module exports an interface while a higher-level client program module can import the interface. This separation is critical for understanding the organization of the Pilot system and its kernel/manager pattern, recalling the policy/mechanism separation initiative of <<cite Levin1975>>. The article describes interfaces for files, virtual memory, I/O, network communications, and debugging. Following this, it describes the implementation of these interfaces according to the organization of client module components and in reference to the manager/kernel design pattern.
There are several novel operating system design aspects that are contributed in this article, many of which are attributed to the purpose of using the operating system on a personal computer. (1) Pilot is single-language system that can be interconnected with a network of other personal computers using the Pilot system, facilitating file and volume transfer with unique Pilot identifiers. (2) A separation of interface (low-level mechanism) and implementation (higher-level application policy), extending the work of <<cite Levin1975>>. (3) Hierarchical spaces in virtual memory for allocating, mapping, and swapping pages, along with advice mechanisms that allow higher-level applications to indicate how virtual memory use can be optimized; this predates similar support provided by the LRU-SP policy in ACFS (<<cite Cao1996>>). (4) Indirect I/O device access via //streams//, mitigating the inconvenience of device access through a driver; additional transducer and filter components can serve to transform the I/O data flowing to and from I/O devices. (5) A network of pilot machines intercommunicate via implementations that predate and implicitly hints at the //end-to-end// argument of <<cite Saltzer1984>>, wherein "it is the responsibility of the communicating end processes to agree upon higher level protocols that provide the appropriate level of reliable communication" (p.86): reliable processing of transmitted data is the client program's responsibility, and that this processing could draw upon common set of features, while also incorporating customizable client-level modifications (p.87). (6) An integrated debugging environment that saves the system state in the event of a crash, allowing the user to operate in a safe recovery mode. (7) An organization of implemented interfaces into component modules, a layering of abstraction toward Pilot clients. In total, Pilot appears to integrate these interfaces and implemented components in a coherent and organized fashion, thereby permitting flexible use and widespread adoption.
For a paper about an operating system for a //personal// computer, I felt that the //person// was oddly absent. The article enumerates the interfaces and implementation modules of Pilot, while distinguishing Pilot from other operating systems, however I was left wondering: why does the individual user need these implemented components? I can't tell if the novel contributions of the paper would also serve benefit the design of multi-user operating systems. Perhaps a better question is why does one require a personal computer? How are the use cases different or similar between a personal and multi-user computers? Given the article, I am lead to think that the authors hadn't considered differences in usage style, that the use of a personal computer running Pilot was implicit, similar in all ways to the use of a multi-user computer, albeit without a need for significant amounts of protection. It appears as though most of the features that distinguish Pilot from other operating systems are transparent to the user, occurring below the client module component level. This recalls an earlier concern that I raised about the design of Hydra (<<cite Saltzer1974>>), wherein it seems that designers of Hydra (and Pilot) do not deem it necessary or important to ensure that the user's mental model of the system matches the actual underlying system model. I conclude with the question of whether it is necessary for users to be aware of an operating system's complexity.
!!!!Discussion Points/Questions
*In 1980, did system designers expect use cases to be different between a personal and multi-user computers? What impact did this have on the design of operating systems for early personal computers?
*Pilot "provides a basic set of services". Pilot "omits certain functions [from] other operating systems". Pilot "provides a more complete set of services than is normally associated with the kernelﴻ (p.81). Is it possible to be both basic/minimal //and// complete? Which is Pilot?
*The authors envisioned a strong coupling between an operating system a network, that a single network could only be populated by machines using the same operating system. How widespread was this idea in 1980 and when did attitudes change?
*What is the difference between a //defensive// and //absolute// protection mechanism (p.81)?
*What are the language-independence arguments that "are not relevant" in the case of Pilot (p.82)?
*Why does the page granularity become too coarse at lower levels of suballocation (p.84)? Wouldn't it become too fine?
*Other parts of the article that I found confusing or unclear / difficult to understand / I lack a sufficient background to appreciate:
**The UNIX pipe and filter facility and how it resembles/differs from Pilot's transducer and filter facility (p.85).
**The three levels of packet protocols provided by Pilot (p.86)
**Pilot's trap handling (p.87).
**Dedicated front-end communications processor, and the need for them in other operating systems (p.91).
!!<<cite Lauer1979>>: Duality of Operating Systems
This article, a collaboration between Xerox and Cambridge University, was published in 1979. The timeline of other papers read in this course served to put the argument of this paper into perspective: prior to this article, we are aware of two very different operating system structures: <<cite Saltzer1974>>'s Multics and <<cite Wulf1974>>'s Hydra, which were message-oriented and procedure-oriented, respectively. We also read <<cite Levin1975>>'s paper on the separation of policy and mechanism, which, in principle, allows for the design of flexible and custom policies at higher levels of abstraction. While this separation may seemingly give way to radically different system structures, Lauer and Needham's article argues that the two major families of system structures, these being message-oriented and procedure-oriented systems, operate in a semantically similar fashion. Furthermore, the operations of a message-oriented system can be expressed in using the syntax of their dual operations in a procedure-oriented system, and vice-versa. In order to show this, the authors first characterize both system structures and their respective programming styles. They proceed to illustrate the duality of message-oriented systems with procedure-oriented systems by mapping the syntax of one with the other, assuming certain system architecture equivalencies.
*Aspects of ''message-oriented systems'': specific and explicit communication paths, a static number of processes, static contexts / dedicated memory spaces, resources; message queues, data structures passed by reference, I/O treated as virtual processes, static priorities, small number of messages, no global naming; messages and message identifiers, channels and ports, processes and process declaration;
*Aspects of ''procedure-oriented systems'': locks and lock releases - locked resources have small locality, global data access and universal naming, easy process creation/deletion, processes wander (like Hydra) and have dynamic priorities, processes make synchronous and asynchronous calls (fork and join) to other processes, modules and module instantiations, monitors and condition variables (wait, signal), modules not deleted
*The duality mapping between message-oriented systems and procedure-oriented system can be applied in both directions:
**process = monitor (the most interesting mapping)
**create process = new /start monitor
**message channels = external procedure identifiers
**message ports = entry procedure identifiers
**send message; await reply (immediate) = simple procedure call
**send message; await reply (delayed) = fork
**send reply = return (from procedure)
**main loop of standard resources manager, wait for message statement, case statement = monitor lock, entry attribute
**arms of case statement = entry procedure declarations
**selective waiting for messages = condition variables, wait, signal
The authors base their argument on accumulated empirical observations, providing as evidence a case study of the Cambridge CAP computer, whose system components underwent a message-oriented to procedure-oriented transformation. While quantitative system performance data was not reported, we are to assume that the original system performance was preserved following the transformation. The focus of this case study was instead directed at the ease of the transformation itself. Unfortunately, many system architectures are less flexible than that of the Cambridge CAP, and few such transformations have been documented. Accordingly, it is the system architectures that dictate system structure, either message-oriented or procedure-oriented. The authors observe that system architectures are often decided upon as a result of misguided organizational policy and a lack of a precise, consistent terminology. Such decisions limit the types of systems that can run on these architectures, as well as the potential for transforming a system of one type to another. As a result, the authors advocate a careful consideration of system architecture, the desired operating system structure, and whether such a system could afford a future transformation.
The paper was clearly-written, and did not assume much in the way of prior knowledge, which I appreciated. While I imagine that some readers may find this paper to be too basic or verbose, I found that it helped to solidify much of what we've discussed over the past several weeks (throughout my discussion jottings I have listed several modern system analogues of message-oriented system components - this article helped to confirm these).
!!!Discussion Points/Questions
*an ''easy read'': clarity of writing vs. prior understanding; practical, straightforward in tone
>MB: //"I found that it helped to solidify much of what we've discussed over the past several weeks (throughout my discussion jottings I have listed several modern system analogues of message-oriented system components - this article helped to confirm these)"//
**flip side: obviousness? hand-holding? verbose? trivial? hand-wavy? not useful? oversimplifying?
***argument is "elegant though not deep enough to be convincing"
>JC: //ﴻanything that can be accomplished by sending a message to another process and waiting for its result, can be done by calling into that code directly, something that seems to be extremely trivial point to write a paper about."//
**Ongoing vocabulary discussion / analogues: process vs. thread vs. computation vs. monitor - making shared semantics explicit
*''timeline'': collab. b/w Xerox and Cambridge in 1979. after <<cite Saltzer1974>>'s Multics and <<cite Wulf1974>>'s Hydra (message-oriented, procedure-oriented, respectively). After <<cite Levin1975>>'s paper on the separation of policy and mechanism. Before <<cite Redell1980>>'s Pilot/Mesa (Xerox), before <<cite Saltzer1984>>'s end-to-end argument, <<cite Clark1985>>'s upcalls paper.
*(meta-level) empirical vs. subjective / opinion - position paper / philosophical (no objective performance data)
*The authors perceived their argument to be controversial - is this still the case? "Time will tell" as to whether they are correct - has it?
**distributed systems, networked systems, modern machine architectures
**is the argument still relevant? dated?
***modern OS: monolithic kernels w/ procedure calls w/in them, application level processes communicate via shared memory / message passing
***where does UNIX fall on the spectrum (mixed membership)? Microkernal OS?
*Is it useful/practical/necessary to classify systems in this manner?
**transformations reduce to easy and uninteresting research problems
*''contribution'': duality argument as a guidelines for evaluating systems a priori / avoiding misguided organizational decisions
!!!!Canonical message-oriented systems
*e.g. GEC 4080, Multics
*What are transactions (in real-time message-oriented systems)?
*What makes the GEC 4080 so elegant?
*Do newly created processes in message-oriented systems get their own new ports? Why is delete process messy and unimportant?
!!!!Canonical procedure-oriented systems
*e.g. Pilot/Mesa, Hydra
*In procedure-oriented systems, why is //delete module// not provided?
*Re: similarity of programs, why does ''wait'' provide a richer synchronization than does selective waiting for messages? How does ''wait'' ensure that the monitor invariant is true?
*purpose of the ''entry'' attribute?
!!!!On the duality
*''main contribution'': observation that neither approach is universally "better"
**understanding this equivalence reduced strong (irrational?) belief in superiority of a system style
**differences are syntactic, not semantic (logically identical, semantically invariant) - bulletproof in a fully dichotomous/canonical world? - everything else (hardware) being equal
**<<cite Lauer1979>> contradicts previous papers; <<cite Clark1985>>: "message passing always results in poorer performance than procedure calls"
>RS: //"flies in the face of the loudly expounded benefits of systems we have read about"//
*''take-home message'': design not for high-level apps, but for low-level architecture, which dictates mechanisms and therefore policy
>NT: //"As hardware design has settled down I wonder if it is the case that we just as easily could have ended up with message passing being the dominant communication structure, or if there are reasons that make the supporting hardware less preferable than hardware supporting function calls at the same execution speed."//
*two systems not as equivalent as claimed; do hybrid/compromise systems have poor performance? anecdotal evidence only?
**The quoted open question in the //Similarity of Programs// section:
>//"would a style/mechanism that has no dual be considered truly-well structured construct that is elegant in form and rich in semantic content? or is it overimaginative, ungainly, awkward to program and hard to understand?"//
>
>RS: //"capabilities of either approach that do not have a dual are by nature convoluted and should be considered a liability rather than a feature"//
*logical proofs to show equivalence of two systems? (if a claim of logical equivalence is made, should a rigorous proof follow?)
*Authors biased towards one set of primitives (i.e. Mesa-style procedure-oriented model)
**strongly-typed languages - textually identical except for keywords
!!!!Assumptions
*"friendly machine architecture": architecture dependencies and caveats b/w two approaches
*identical performance re: primitive system calls
!!!!Evaluation / The Cambridge CAP case study
*performance observation and low-level metrics: time, computational overhead, queuing / wait times (efficiency) - time most important support for comparison argument
**overhead of message passing (queueing and copying) 쬩ng into a procedure (monitor enter/exit - <<cite Clark1985>>)
**worse-case vs. best- or average-case comparison
*Authors admit difficulty re: changing the structure of operating systems to reflect the duality that they suggest
*I had a difficult time understanding the paragraph on the Cambridge CAP computer's architecture. I could use an explanation of this.
!!!!Org. policy
**organizational policy, human factors considerations - high-level tasks dictate low-level architecture choices, which limits OS/kernel development
!!!!Other
*What is meant by //system internal names//?
*What is meant by //zeroeth / first / 2nd order considerations//?
!!<<cite Engler1995>>: Exokernel
I'll begin with the caveat that I found Engler et al.'s article to be incredibly dense, at least relative to the other papers we have read in this course, and while I think that I might have a conceptual understanding of the authors' contributions, I make no claims to an understanding of its more arcane details. I can often reliably attribute my misunderstanding of a paper to either the clarity of exposition or to my unfamiliarity with the subject matter, however in the case of this paper I can't discern the relative extent to which these factors hindered my reading. With that being said, I'll take a stab at summarizing and reviewing it for the purposes of our discussion.
An exokernel is an interface that makes hardware resources available to untrusted applications without compromising the safety of the entire system or impacting the performance of other applications. This gives applications the flexibility to use resources as they see fit, optimizing their use based on their unique needs. This recalls application-controlled pre-fetching and caching (<<cite Cao1996>>) and our seminar discussion of applications with unique memory access patterns, such as streaming video. In short, an application's intended resource usage pattern is more informed than a monolithic OS-wide policy, which can hurt application performance and limit individual application functionality. The difficulty with application-level resource meanagement is that applications are untrusted; giving unrestricted access to resources compromises system security.
Engler and colleagues have proposed a way to separate resource access (management) from protection. Available resources are //exported// using //secure bindings//, or secure access points, visible to applications by name and tracking resource ownership. Resources can be //revoked//, however this too must be visible to applications using these resources. Uncooperative applications that do not relinquish revoked resources can have their resource access shut off via an //abort protocol//. Aside from these protection measures, applications are free to use the resources as they see fit. The applications themselves run in dedicated application-level //library operating systems//, which are in themselves quite extensible.
The authors go on to describe their implementation of an exokernel in great detail, and report on an exhaustive experiment to compare the performance of an exokernel (Aegis) with a library OS (~ExOS) against that of a monolithic operating system (Ultrix) on several of machine architectures. The results of these experiments showed that an exokernel can be implemented efficiently, resource ownership tracking is simple, and applications' data structures can be kept in physical memory. Only in the case of batch protection changes to pages was Ultrix faster than ExOS; in all other aspects of virtual memory management, ExOS outperformed Ultrix. Adding Application Safe Handlers (ASHs) to ExOS was found to boost performance; the result was striking (see Figure 2): roundtrip packet latency remained constant in the ExOS+ASH variant as processes were added, whereas in the ExOS-ASH variant it increased as more processes were added. This interesting result was somewhat muted. More generally, I felt that the overwhelming volume of experimental results presented in this paper made it difficult to tease out those which were most practically significant; none are emphasized and thus individual results are difficult to retain after reading the paper (they are also difficult to keep track of while reading the paper!). In the end, individual results are not remembered, only the general finding that ExOS outperformed Ultrix is retained.
Performance differences reported in this article are descriptive without indicating as to whether they are practically significant or noticeable in practice. I had a similar question relating to the performance measurements reported in the <<cite Cao1996>> article. The authors do not report the variance of these measurements, or how these pooled measurements were distributed prior to their aggregation. I'm surprised that "ExOS was an order of magnitude faster", for example, is sufficient empirical evidence. I feel that result statements like this require additional context and qualification.
!!!Discussion Points/Questions/Aspects needing explanation
*On p.3 final para of section 2, what is meant by backward compatibility?
*On p.5 (Multiplexing the Network), what is demultiplexing?
*Section 3.2.1: my understanding of how //application safe handlers// work to download application code into the kernel is hazy at best. I could use a dumbed-down explanation of what's going on here - how can these access application data? via upcalls?
*Re: dynamic code generation - wasn't this discouraged in <<cite Dennis1966>>? Is it permissible here because a single application's resources will not be shared with other applications?
*Are the performance differences mentioned in this article practically significant / noticeable in practice?
*Is it common in more modern OS research papers (1990s+) to discuss related work at the end of the paper?
!!<<cite Kaashoek1997>>: Exokernel follow-up
In this follow-up to their 1995 Exokernel paper, Kaashoek and colleagues reflect further on several years' experience building an exokernel systems, library OSes, and library file systems. As a result, I feel as though I now have a more complete picture of how each of these interact, since the two articles complement one another in terms of the details that they contain.
The article begins with a motivation for exokernels: providing applications the freedom to manage their own resources in a protected manner. This motivation is familiar, and was addressed earlier by the ACFS and LRU-SP policy of Cao et al. (1996). However Kaashoek and Engler's approach does not involve a single monolithic operating system and file system. Instead, the exokernel approach involves a small kernel and a set of extensible library operating systems, customized for each particular application's unique needs. However, this design brings about questions concerning application performance as well as global system performance. Related work has considered this question, namely work pertaining to microkernels and virtual machines. Regarding the former, the authors explain regardless of whether a system is monolithic or microkernel-based, the issue of how the application controls resources remains. Regarding the latter, virtual machines emulate, which hides information and thereby resources from applications. On the other hand, exokernels expose resource information about resources via access control and secure bindings. How the applications make use of those resources is decided upon by the library OSes that contain them.
The article discusses how storage is multiplexed with the Xok exokernel and the XN storage system and the C-FFS library file system (a UNIX-like file system). Some of the defining characteristics of this storage system include:
*Library file systems can easily create files and file formats, and share them with other library operating systems and file systems.
*Application defined file metadata layouts are translated by untrusted deterministic functions, allowing the kernel to handle different metadata layouts without understanding them (I cannot grasp the full complexity of how UDFs operate)
*XN allows mutually distrusting applications and their library file systems to share metadata, and maintains a registry of cached disk blocks. It ensures that access to these disk blocks is consistent and coherent, preserving the integrity of these file systems.
*XN includes utilities for creating new types described by library file systems, making these file systems persistent, as well as reading, writing to, allocating and deallocating blocks from disk.
The following section of the article describes the combination of Xok and ExOS. Xok replaces Aegis, the exokernel of Engler et al. (1995). ExOS, the library operating system that can incorporate UNIX abstractions, was also described in that earlier paper. Unlike Aegis, Xok runs on a different hardware architecture, includes the ability to inject application wakeup predicated into the kernel, as well as hierarchically named capabilities for access control. Does this access control limit the flexibility of application-level access control?
The remainder of the article demonstrates the performance and extensibility of Xok/ExOS using the C-FFS file system, to which I had several comments and questions. With regards to single-application performance, I was curious as to how the authors determined what was to included in the I/O-intensive benchmark. I was intrigued that Xok/ExOS runs "considerably" faster for some applications, though the authors do not know why (ᬠp.9). It appears that some unforeseen interaction is occurring, and I wonder if the authors have since determined what this may be attributed to. Finally, given that the goal of the exokernel is to preserve global system performance for multiple concurrent applications, I was surprised that it had not been extensively studied. Once again, the experiment that they report in 쥦t me wondering as to whether the methodology produces an externally valid result; they claim to offer a "useful relative metric", but I'd be curious to see how this methodology generalizes. They select multiple applications that compete for resources, however I'd like to see how this compares to accumulated observations of how such systems are used, by real users in real-world contexts.
I'm beginning to notice a general trend with the performance evaluation work that we've read. As I'm more familiar with experimental methodologies in other areas (cognitive sciences, HCI), I remain somewhat puzzled when I come across implied or ambiguous hypotheses, partial or missing rationale for choices of low-level metrics (e.g. reporting minimum vs. average run times), and the use of phrases like "considerably faster" or "significant improvement". Is a science of system evaluation still emerging, or does the community not perceive a need for a standardization of how these evaluations are conducted and reported?
To conclude, I found this paper easier to digest than the earlier Engler et al. (1995) paper. Although there is still much that I don't understand, I felt that this paper was more approachable and provided a better big-picture perspective on the issues at play. In particular, I found the final section to be quite helpful for summarizing both the advantages and disadvantages of the exokernel approach, the lessons the authors have accumulated, and the need for further study with respect to simplifying exokernel interface design and handling information loss.
!!!Discussion Questions:
*Does Xok's access control, intended to simplify the implementation of secure applications, restrict the flexibility of these applications? Does this design choice place too much policy at the kernel level?
*Re: evaluation measures, how and when are previously published workload benchmarks selected? How can we assess the applicability or appropriateness of these benchmarks in a given situation? (I imagine some of this may be found in [33], for instance). Are benchmarks intended to be used to simulate worst-case or average-case scenarios?
!!<<cite Bershad1995>>: SPIN OS
The article by Bershad and colleagues presents an operating system known as SPIN, designed using an alternative approach to that of monolithic kernel, microkernel, and exokernel systems. The motivation for this work is familiar to us from other articles read in this course (<<cite Cao1996>>, <<cite Engler1995>>): applications use system resources in different ways, and one-size-fits-all global resource policies do not permit optimal application performance. This article succinctly points out that the solution to this problem requires a compromise between safety, performance, and extensibility. Comparatively, monolithic kernel systems are not extensible, microkernel systems perform well but do not provide fine-grained specification of how resource interfaces are accessed, and exokernel systems separate resource management from protection abstractions provided by the hardware, thereby making protection difficult to control.
In the SPIN system, Bershad and colleagues demonstrate that applications can make use of modular kernel extensions for memory management and thread scheduling. These extensions provide access to system resources via interfaces defined in protection domains. Dynamic linking permits extensions to communicate efficiently with one another and kernel core services; this does not incur a high communication overhead, as linked domains can communicate directly. Extensions are written in Modula-3, a type-safe language and a descendant of the Mesa language used in the Pilot system <<cite Redell1980>>. Unlike Pilot, SPIN systems are not limited to a single address space or programming language environment. Applications access protected resources via capabilities, which are defined as pointers in the Modula-3 language.
The latter half of the article compares the performance of a SPIN system against a monolithic kernel system and a microkernel system. What wasn't clear to me about this performance evaluation was the extent to which extensions were used; I can't tell if all SPIN microbenchmark measurements relied upon the extensions to memory management and thread scheduling described in the preceding sections; would a SPIN system without extensions perform just as well as a monolithic kernel or microkernel system? If the core system services differ substantially from these other systems, and I suspect that they do, I would have expected a SPIN-without-extensions variant to serve as a baseline for performance comparisons. Regarding the microbenchmarks themselves, can these results be extrapolated to global system performance? Can they account for multiple concurrent applications? Are these microbenchmarks a suitable granularity for comparison? I was expecting both a local and global performance comparison, something akin to the evaluation described in <<cite Kaashoek1997>>. Finally, I appreciated the end-to-end performance evaluation of a networked video system, a tangible application. Despite this application example, I was still uncertain as to who SPIN was designed for; can SPIN be used on a personal computer, such as <<cite Redell1980>>'s Pilot system, or is SPIN's extensibility overkill for anything but dedicated server installations?
!!!Discussion Questions:
*Re: మ 276: short of reading [Anderson et al. 91], why can't benchmark measurements be scaled to facilitate comparisons across different hardware architectures? Is this why a comparison between SPIN and an exokernel system was not attempted?
*The authors report the size, in lines of code, of system components and extensions. At what point does the size of system components and extensions become too small and thereby difficult to use and understand by those who make use of them?
!!<<cite Pike1995>>: Plan 9
//Plan 9 from Bell Labs// differs in several ways from the other papers that we have read to date. It is an odd mix of a position paper and a high-level overview of a computing environment, reading more like the introduction to a Plan 9 usage manual than that of a research paper. As a result, it is more straightforward and accessible than other contemporary papers, especially those addressing particular research questions about extensibility and performance (e.g. <<cite Bershad1995>>, <<cite Engler1995>>). On the other hand, Pike and colleagues have described Plan 9 as a platform for their research, but they have not described the research process itself. Given their motivating position that personal computer systems "fractured, democratized, and ultimately amplified administrative problems", it seems that Plan 9 was poised to serve as a vehicle for studying these higher level organizational choices. They state that the true test of the system is the "computing environment it provides", whether it encourages a "more effective way of working". Unfortunately these questions go unanswered; I would have enjoyed reading about a longitudinal study of how workers use a Plan 9 installation over time, indicating whether or not work practices change. Instead, the paper focuses on Plan 9's engineering details.
Plan 9's design does include many novel features, however the details of these features do not coherently and explicitly relate back to their motivation for changing work practices. Nevertheless, many of these design features are familiar to users of thin client terminals connected to a central server, with a client's view of the system remaining customized and constant on different terminals. I found their use of analogy and the phrase "my house" is helpful for explaining this. Several other features of Plan 9 are novel and worth noting:
*All resources and objects in a Plan 9 installation are described like files in a hierarchy, a design which the authors argue to be easy to understand by system developers and users alike.
*The system was not designed to be compatible with old commands or programming notation that were seen to incompatible with the motivation of the system.
*The system can be easily restored from daily backups using a write-once-read-many bulk storage system.
*Users interact with the Plan 9 installation via the 8鮤ow system, a view into the client's private name space but also into remote FTP sites and CPU servers. Files are typically not cached on the client.
*Plan 9 is portable, running on various hardware architectures. An implication of this design decision is that terminals don't need to be replaced as often as central CPU servers, thereby a cost-saving measure.
*Plan 9 flexibly supports parallel programming in Alef and C.
*Most transactions are direct procedure calls, only mounting an external file server requires a costly remote procedure call.
*There is no super-user. Individual users are responsible for their own files. However there exists roles for sysadmin maintenance users and anonymous users. Groups of users are permitted, and group leaders are optional. Hierarchical user roles recalls the discussion from Saltzer's 1974 Multics paper relating to whether a "locksmith" user is needed.
The paper concludes by offering a simple performance measurement, demonstrating that Plan 9 can perform common low-level operations comparably well against competing commercial systems. Granted, if I represented an organization considering a Plan 9 installation, I would rely upon such performance measurements. However as a researcher, it's not clear what the novel research contributions are at this point. I had a similar reaction to the early Multics and Hydra papers, which proposed systems without implementing and evaluating them. At least Wulf. et al's Hydra paper contained an imagined use case describing how the system could be used. I felt that this paper could have taken a similar approach, describing a use case which addressed their motivating problem, indicating how work practices could differ with a Plan 9 installation.
!!!Discussion Questions:
*Did Plan 9 deployments succeed? Why or why not? Did it save money?
*Since compatibility was not a priority, were initial users or developers confused or frustrated by the lack of familiar commands and notation?
*The paper doesn't discuss tradeoffs between extensibility, local and global application performance, and protection, which have been the emphasis of other papers read recently. Where was Plan 9 situated within this tradeoff space?
!!<<cite Liedtke1995>>: microkernels
Liedtke wrote in defence of microkernels. Prior work had proclaimed microkernels to be inefficient and inflexible due to address-space switches and a dependence on operating in user-kernel mode. He claims that while some performance measurements have shown microkernels to be inefficient, researchers have not investigated the causes of these inefficiencies; as was the case in <<cite Kaashoek1997>>'s article, which demonstrated that the Xok exokernel performed more efficiently than the Mach microkernel on several microbenchmarks.
Liedtke goes on to describe the minimal core services of microkernels, and explains how the shortcomings of other microkernel designs can be attributed to improper implementation of these core services or from overloading the kernel with more mechanisms beyond these core services. Kernel-user mode switches can be implemented to be more efficient by making use of a kernel stack per thread and supporting persistent user processes. Similarly, address space switches can be more efficient by using tagged ~TLBs. The decision to use a segment-based address space, rather than a page-based address space, can improve IPC. Reconsidering the size of the cache working set can improve memory performance. Additionally, he maintains that microkernels must maintain a tight coupling with the computer's architecture, and are therefore not portable to other architectures. A portable microkernel requires an additional layer of abstraction between the hardware and itself, which reduced efficiency; therefore some previously reported poor performance may be attributed to this issue of non-portability.
Liedtke's own microkernel, L3, was designed according to the rationale in this article. This rationale puts functionality and flexibility before performance, and by providing a minimal set of primitives shared by all applications, the resulting microkernel performs comparably well against monolithic kernel systems. For protection, the only policies that the microkernel must support are the principle of independence and integrity between arbitrary subsystems. For sharing, it must support granting, mapping, and flushing pages from address spaces - though the creation of these address spaces and their paging policies is not the jurisdiction of the kernel. The microkernel supports IPC between address spaces, but not the form of IPC. I/O, hardware interrupts, and remote communication are not handled by the microkernel; each of these can be flexibly specified at a higher level. The overarching message here is that a microkernel probably doesn't need to provide as many abstractions as one would think.
I found this article to be helpful for understanding the nuances and differences between microkernels, exokernels, and extensible operating systems, such as <<cite Bershad1995>>'s SPIN system. In microkernel systems, the operating system is a user-level application that describes all policies based on a minimal core set of kernel abstractions. Exokernels do not provide abstraction and can support multiple library operating systems, however the kernel is still responsible for protection policies. The SPIN system allows the operating system to extend kernel mechanisms and system calls, and operate in kernel mode when it needs to, to compiled type safe extensions. It is now clearer to me how these represent different approaches to the problem of efficient and flexible operating systems.
The clarity of writing was higher than that of the exokernel and SPIN articles, and I appreciated the format of each section: first proposing a previously reported problem, an examination of the cause, a logical argument defending the microkernel concept - placing the blame on inflexible implementation or incompatible hardware, and a concise summary. The description of other systems in 硳 also helpful for comparing between different approaches to small kernel design. Nevertheless, there were still some sections which discussed topics that I was unfamiliar with, and I hope that discussion can help to unravel these.
!!!Discussion Questions:
*Do efforts to write portable microkernels continue today, despite Liedtke's argument? Have modern architecture designs converged or do drastic differences in architecture persist, hampering portability?
*Re: interrupts in ⠨p.240), how is interrupt handling "hidden by the kernel"?
*Are exokernels more portable, given that they provide less abstractions to library operating systems? The performance evaluation of Xok/~ExOS in <<cite Kaashoek1997>>'s paper compares against monolithic operating systems, but how would it compare against L3 on the same hardware? (Or would this be akin to comparing apples and oranges?).
!!<<cite Chase1994>>: ~Single-Address-Space OS
This is another article from the folks at U. Washington, which brings together many of the concepts we've discussed to date: the separation of protection from resource management, segment-based addressing, and modular code structures. It was published around the same time that other groups were anticipating the development of applications with unique resource needs and trust relationships, thereby necessitating flexible and extensible operating system structures and application-specific policies. In their article, Chase and colleagues observe another growing trend in application design: inter-application integration and cooperation, by way of sharing of complex data structures between applications. Additionally, these applications may have asymmetric trust relationships. The authors make the case that this inter-application integration is poorly supported by other operating system designs, particularly those that make use of private address spaces for each procedure. As a result, other systems compromise either security or performance. Chase and colleagues propose a solution, one that hinges upon the recent appearance of 64-bit address space architectures, by allowing all procedures to share a single address space. This design permits a full separation of addressing and protection, no longer requiring costly context switches and virtual address translations, ensuring that pointers to addresses are context independent, remaining valid across protection domains.
The article describes the concepts and abstractions of a system-address-space operating system. These are largely transparent to applications, which cooperate via mediators rather than at the level of these abstractions. Some of these abstractions are familiar to us, recalling early Multics and Hydra papers: segments, capabilities, and access control lists. I was initially confused by the inclusion of both capabilities and access control lists, as I thought this was an either-or design decision. But after reading further, I realized that the two forms of security are complementary, that resource access can be denied in specific contexts even if the requester holds the capability for that resource. Interdomain communication is handled similarly via portals, an abstraction of entry points into a domain. I was initially surprised that resources were described at the level of segments, given the later case study of applications sharing larger data structures such as objects and files. Upon further thought, I realized that there may be situations in which intermediate results or smaller-scale elements of these objects may be required by cooperating applications. Though this case isn't described explicitly in the case study, I imagine that sharing data at a finer granularity than objects or files is often necessary, particularly when applications describe objects using different languages and using different levels of detail.
I found the section on implementation to be the least straightforward section of the paper. I was puzzled as to why Opal was not implemented as a monolithic kernel system, as opposed to running above the Mach microkernel while also leveraging a Unix server. I felt that demonstrating the implementation of Oval with Mach was a second, yet separate contribution of the paper. This could have been a second paper, one demonstrating the portability of Opal and the flexibility of Mach. As a result, the current article asks a lot of the reader: first one must grasp all of the abstractions described in d then map these abstractions to those in the Mach microkernel and Unix server in `second paper could have also compared the performance of Opal running natively against an implementation running on Mach. Given the criticism around Microkernels an Mach discussed by <<cite Liedtke1995>>, I wonder to what extent can Opal's performance be attributed to the Mach implementation.
I applaud the authors for describing the integrated Boeing design system: a tangible, real-world case study of a suite of applications that would benefit from an Opal system and the use of mediators. Examples like this make learning the abstract concepts of novel operating systems much more concrete. It also demonstrates the applicability of theoretical and experimental system designs to real-world problems.
The performance results indicate that the Opal system can perform comparably well, in line with unsafe Unix processes sharing a protection domain, while maintaining a level of protection that separate protection domains afford. While these performance results were fairly low-level, I was hoping for some indication of performance at the level of the Boeing CAD system; how long did it take for integrated applications to share data structures of varying sizes? Perhaps the authors, with the help of Boeing engineers, could have determined some application-specific benchmarks.
The rest of the article describes some issues caused by using a single address space, such as the loss of virtual contiguity or the flexibility to use context-dependent addressing. In defence, the authors point out that the size of 64-bit address space and the willingness to break old programming habits (forking, context-dependent addressing) will solve many of these problems. From what I gather, data copying is still an unresolved issue. The authors go on to compare Opal to other systems, wherein many of these issues are unresolved or addressed in such a way that forces a single language model, or otherwise compromises either security or performance.
!!!Discussion Questions:
*How would the performance of Opal implemented as a monolithic kernel compared to that of its Mach implementation?
*There's a note that ~ACLs were not yet implemented at the time of publication; did this layer of protection, when added, impact performance?
*Would application-specific benchmarks be a compelling performance measurement in a case such as Opal, a system designed for application integration and sharing of complex data structures?
*How did descendants of Opal address the data copying issue, particularly when that data contained pointers?
*PA-RISC, as described, was designed to be backwards-compatible with private-address-space systems. To what extent was Opal backwards compatible?
!!<<cite Welsh2001>>: SEDA
As of late we've read and discussed several approaches for providing applications a flexible amount of control over policies relating to inter-application integration and resource usage, with each approach forming compromise between security, performance, and ease of implementation. Observing the rise of internet service usage, Welsh and colleagues point out another issue, that of scalability. An application or service may have many concurrent requests for resources; this is not well handled by thread-based approaches, which can be associated with long wait times when load is high, where not all clients are treated fairly. An optimal scenario for a service with many concurrent requests is graceful degradation, maintaining high throughput while slowing linearly for all clients, maintaining fairness.
Welsh and colleagues propose SEDA, an event-based staged pipeline for routing service requests through components of an application, wherein each modular stage handles a queue of requests. This queue can be monitored such that threads are dynamically assigned or removed from that stage according to its load. The size of event batches sent to subsequent pipeline stages can also be dynamically controlled. They compare the event-based approaches with thread-based approaches, recalling Lauer and Needham's duality argument for procedure- and message-based systems. While a translation is possible between the two systems, procedure/thread-based approaches do not scale to the magnitude that internet services require. It was interesting to see how SEDA blends aspects of both approaches, with threads within stages and events between stages.
A positive aspect of the SEDA architecture is that all load monitoring and dynamic thread assignment is transparent to application programmers, so they don't have to worry about performance tweaking themselves. The authors demonstrate with an implementation of SEDA called //Sandstorm// and two applications, a HTTP server and a ~P2P file sharing service. For the former, they evaluate performance using a web benchmark, comparing against Apache and Flash web servers; they achieve higher throughput, greater fairness, and lower response time with the Sandstorm implementation as more clients requests are made. For the latter, they demonstrate how adding dynamic thread assignment control can keep latency low and event queues short even as ~P2P load increases.
!!!Discussion Questions:
*In the final paragraph they propose SEDA as a new direction for OS design, beyond dedicated systems for internet application services. I was somewhat confused by the mention that "~SEDA-based ~OSes need not be designed to allow multiple applications to share resources transparently", and yet "the system need not virtualize resources in a way that masks their availability from applications". Am I to understand that the availability of resources should be visible, but once they are in use, applications are not aware of the extent to which other applications are using resources?
*In systems where there is no/little concern for scalability, does a SEDA approach needlessly complicate the system design? If the chief concern is performance, would a purely thread-based design be suitable?
!!<<cite Behren2003>>: Capriccio
This is another paper from Brewer's group at UC Berkeley, again addressing the issue of scalability for systems providing internet services. Rather than continuing in the line of SEDA's event-based design, von Behren and colleagues propose a thread-based design solution. They argue that thread-based systems are easier to understand, implement and debug. In the earlier SEDA paper, Welsh and colleagues identified performance bottlenecks associated with thread-based systems; chief among these was that there exists a point where threads have allocated all available address space, thereby slowing the system to a halt. von Behren and colleagues propose a solution to this model which involves user-level threads (as opposed to kernel threads) and dynamic adjustment of threads' stack size, limiting the amount of wasted virtual address space. This is possible by implementing compile- and run-time checks that can identify application-specific needs and blocking points at the various stages of its execution. As a result, the system is scalable and Capriccio applications perform comparably to SEDA applications, with the added benefit that a thread-based approach is less complex.
I noticed that they do not do away with event-based design altogether. In ⬠the authors describe how Capriccio's scheduler operates in an event-driven manner, however this is hidden to the programmer. Yet the scheduler is modular and can be selected at runtime; this seems flexible however at the same time, obfuscated.
Regarding the microbenchmarks, I found that their experimental design was not clearly explained. I'd like to read more about the rationale for the number of trials, the length of trials, the size of tokens, and the number of threads. Do these parameters simply reflect common practice, and if so, is there literature that offers guidance for making these experimental design decisions? How can we be sure that these experimental settings are both externally valid and unbiased?
The dynamic stack management was nifty; their solution combines a compile-time call graph calculation and run-time checkpoints that check for recursion (cycles in the call graph). Thus the stack size grows and shrinks dynamically, albeit in non-contiguous areas of memory.
!!!Discussion Questions:
*Re: modular schedulers, to what extent should event-driven behaviour be visible to application programmers?
*How much rationale is required when using benchmarks of your own design? How can we ensure that benchmarks were not biased to demonstrate superior performance of the current system or inferior performance of prior systems?
*The resource-aware scheduling takes advantage of the fact that all threads are mutually trusting, which is expected in the Capriccio case since the system is likely to be dedicated to a single service or application. Would this scheduling be unfair if implemented in a multi-purpose / multi-application system? Can a system be both scalable and multi-purpose?
!!<<cite Mosberger1996>>: Scout / Paths
There are two major contributions in this article: a definition of the path abstraction, and the demonstration of the utility of paths, both in terms of application performance and the ease of implementation. This article is well situated with respect to our recent discussion of staged thread-based and event-based design of operating systems dedicated to support web servers. In Capriccio and SEDA, application-specific knowledge was required to place bounds on resource usage at different stages of the application's execution. Paths are a related concept, in that application-specific knowledge is used to determine the flow of data between subsystems both before and at runtime, thereby facilitating resource allocation and scheduling decisions. The article also recalls the end-to-end argument (<<cite Saltzer1984>>), since the Path abstraction encompasses the global context of data flowing through a system, from source to sink, rather than on the performance of any particular system component.
Paths rely on early global knowledge of the existence of a message to be sent from one device to another. Devices themselves are abstracted by routers and queues. A path between routers is dynamically created at runtime, based on the current set of invariants describing the state of these routers and their intermittent stages, then optimizing the path according to global knowledge about the system and suitable path transformation rules. Paths are a flexible abstraction, in that they are bidirectional, they can vary in length and width, and can be short-lived or long-lived. I enjoyed how this section (s incrementally described, in their "first principles approach".
The article describes the explicit implementation of paths in the Scout operating system, which runs in a single address space, like the Opal system of <<cite Chase1994>>, and designed for deployment in network appliance devices. The system is thread-based, and it is threads that execute the creation, use, and destruction of paths. As these devices often support a single application, threads are largely cooperative and scheduling decisions are not difficult. A video streaming application is used to demonstrate the utility of the path abstraction, which recalls our discussion of ideal polices specific to such applications, as it is necessary to ensure decent quality of service in terms of output frame rate. They compare Scout's performance against that of a Linux system. Scout outperforms Linux, especially at higher frame rates; however for the average case (most content is shown between 30-60 fps), the differences do not appear to be practically significant. However, Scout shines when the system is under additional load, as video playback on Linux could be considered choppy or jumpy due to the performance degradation. Scout succeeds here due to its ability to segregate types of work, thereby eliminating bottlenecks and the front-of-line problem.
!!!Discussion Questions:
*Re: the network view of paths (㩬 I'd appreciate an example of a weak set of invariants. How could such a degenerate case come to pass?
*One aspect of the paths abstraction that was unclear to me was why it is preferable to have long paths; is this solely for the purpose of accountability? Is the efficiency of short transformed/optimized paths at a lower priority than accountability? If accountability is so valued, then I'm confused as to why intermittent path stages do not also have queues, akin to the event queues of stages in the SEDA architecture. Would this not provide additional accountability and the ability to identify intra-path bottlenecks and the flexibility to create on-the-fly path re-routing?
!!<<cite Pai2000>>: ~IO-Lite
Pai and colleagues describe their attempt to reduce redundant or multiple simultaneous copies of I/O data. Their intent was to design a general-purpose I/O that unifies caching and buffering, such that all subsystems optimally and safely access and share the same physical data. Less copies of I/O data means valuable memory space is freed, thereby improving overall system performance. Their intent was realized in their implementation of the ~IO-Lite API, which can be used transparently by applications already using stdio; this required only slight modifications to an existing ~FreeBSD kernel, their testing platform. Alternatively, applications can use the API directly to take full advantage of the optimizations and flexibility it affords. While some of these optimizations may have been possible in exokernel systems, they required dedicated library operating systems; the ~IO-Lite API is intended to be used in the context of general-purpose operating systems.
The API introduces a model where all sharing of buffers is read-only, permitting safe access to resources between OS subsystems and applications at a page-based level of granularity with access control lists. This involves immutable read-only buffers that are linked together by a buffer aggregate, a mutable abstract data type. When shared, the receiver is given read access to the buffer's pages. This relationship can be made persistent, as when the receiver deallocates the buffer, a producer can temporarily write to this buffer once again, without having to explicitly set up a new sharing relationship with the receiver. This facilitates an efficient producer-consumer relationship. The API also permits copy-free network I/O, making use of a packet filter for determining the designated access control list of the incoming packet, a process known as early demultiplexing. An added bonus of a single network I/O copy is that a checksum calculation need only be performed once, not once for each subsystem the data moves through. Also addressed by the API are customizable cache replacement polices; since buffering and caching are unified, there will be situations where a I/O buffer page is selected for replacement. Application-specific caching polices are customizable, which recalls the approach taken by <<cite Cao1996>> with the ACFS.
The ~IO-Lite API is particularly well-suited for use in the implementation of web servers. In the case of serving dynamic content, memory resources can be scarce. The ~IO-Lite API frees up memory that would otherwise be used for buffering multiple copies of the data. They demonstrate the effectiveness of the API by conducing several experiments, comparing their ~IO-Lite web server (~FlashLite) against Flash and Apache, the former being their current state-of-the-art, event-based, research web server, and the latter being a widely-available and widely-used web server. I found this multi-stage performance evaluation to be well thought out and revealing for two reasons: First, I appreciated the trace-based evaluation (䩠derived from server logs of actual web servers, thereby having a high degree of external validity. Second, the optimization contributions of (橠effectively communicated which aspects of the ~IO-Lite API were responsible for ~FlashLite's superior performance (it appears that checksum caching is essential to this, especially for smaller data set sizes).
!!!Discussion Questions:
*In 䬠how is it possible that ~IOL-read may return less data than requested? It's unclear to me how this may occur.
*Re: worst case modifications to data objects in 謠which uses mmap to create a contiguous mapping of an I/O object that can be modified-in-place. What is the overhead associated with committing the modifications back to a highly fragmented buffer aggregate once modifications are complete? Wouldn't it be less effort to have the buffer aggregate point to a new contiguous copy?
*Clarification needed: the simulation of a wide area network with a number of different clients left me wondering: does ~FlashLite not create a separate process for each simultaneous connection?
!!<<cite Bugnion1997>>: Disco
Disco provides a way to run multiple heterogenous operating systems on the same machine. In describing how Disco accomplishes this, I found it useful to compare Disco to a microkernel and an exokernel, as we've recently discussed both concepts. Like a microkernel, Disco supports varying flavours of operating systems, separating them from the hardware by a thin software layer. Unlike a microkernel, this layer simply mirrors hardware resources and can support multiple operating systems at once, including (largely) unmodified commodity operating systems. An exokernel can also support multiple library operating systems, again separated from the hardware resources by a layer of indirection. The difference, however, is that Disco does not require these to be custom library operating systems, and it "virtualizes rather than multiplexes" the system resources. It took me a while to wrap my head around this difference but now I think that I have a better grasp of it: Disco's sharing of physical resources across a set of operating systems is transparent to any one of them. Disco provides a translation layer that gives each operating system the impression that it is running on its own dedicated machine; each is a virtual machine. This means that multiple unmodified commodity operating systems can exist simultaneously. The appeal of this organization is that faults can be contained to a single virtual machine without affecting the others.
Disco can be described as a virtual machine monitor, which is a design approach dating to the early 1970s. The novel contributions of this work are (1) the extension of this virtual machine monitor approach to the new generation of scalable multiprocessors; (2) overcoming memory and execution overheads associated with hosting multiple virtual machines by allowing the sharing of a global buffer cache, making use of copy-on-write disks and a policy of dynamic page migration and replication; (3) communication and sharing between these operating systems is handled by existing distributed networking protocols, much like as if the systems were on physically isolated machines. Unnecessary copying of data is eliminated (much like in ~IO-Lite), since shared data between virtual machines can refer to the same machine pages, made read only in the physical pages of the server and the client. Prior to Disco, resource management, sharing, and intercommunication between virtual machines was inefficient, with each virtual machine competing for resources.
Cache-coherent non-uniform memory architectures (ccNUMAs) are a powerful type of multiprocessor that, in 1997, weren't being exploited by commercial operating systems. While incredibly scalable, memory management with these architectures is a nasty problem, requiring operating systems to accommodate for this. Instead, Disco handles memory management, allowing existing unmodified commodity operating systems, not initially designed for ccNUMA architectures, to enjoy the benefits of its scalability. Disco gives each OS its own "physical" address space starting at address zero, a transparent abstraction layer that maps to the actual machine address space. It also virtualizes I/O, disks, networking, and ~CPUs. It maintains a 2-level tagged TLB for each CPU, with physical to machine mappings in the first and recent translations in the second, so context switching is fast. Additional performance gains are the result of dynamic page migration and replication: in response to cache misses (which are particularly expensive on NUMA architectures), the locality of virtual machine memory access can be improved when multiple virtual ~CPUs map the same physical page; a machine page can either be migrated or replicated to the machine address space of the other virtual ~CPUs. This is transparent to the virtual machine, which still refers to the same "physical" page.
Commodity operating systems can be supported by Disco upon performing minor tweaks to their hardware abstraction layer. Many of these tweaks are responses to the peculiarities of MIPS nodes on the FLASH ccNUMA multiprocessor. The authors demonstrated this tweaking by porting Silicon Graphics' IRIX OS to run on Disco. Specialized application library ~OSes (like those built for exokernel systems) that scale with the architecture can also be supported by Disco. The authors developed one of these as well, SPLASHOS, intended for dedicated scientific applications.
They present experimental performance results, comparing varying numbers of virtual machines running on Disco against a standalone IRIX system. They were unable to use a FLASH ccNUMA, which was not yet available at the time, so they ran their experiments on ~SimOS an architecture simulator. I discovered that ~SimOS was developed by the same research group. I wonder how widely available it was. I took them at their word for how they configured ~SimOS, but I have to wonder about the reproducibility of this research. Could they not have waited until FLASH, a commercial 3rd party architecture, was available? I'm uncertain of these results, as they are akin to taking length measurements with a ruler of your own making. Since FLASH was unavailable, they were additionally unable to answer questions regarding long-running workloads and resource sharing between virtual machines. This is disappointing given how these virtual machines may be used in practice, potentially intercommunicating over long periods of time. Nevertheless, they examine measures of scalability, memory and execution overhead, and memory management. I feel that many experimental design details were either ambiguous or glossed over: the number of trial runs, rationale for the number of ~VMs, and the homogeneity/heterogeneity of multiple ~VMs. I have a number of comments and questions regarding these experiments, which I list below.
This was not an easy paper to read. In addition to being overly verbose, there were many occasions where I found myself referring to Wikipedia for defining unexplained concepts. It took a lot longer than usual for the big ideas in this paper to sink in.
!!!Discussion Questions:
*Could they not have waited until FLASH, a commercial 3rd party architecture, was available? Was this publication premature or were they afraid of being scooped?
*Re: execution time overheads, is up to 16% additional overhead really considered "modest"?
*Their experimental protocol doesn't explain how did they decided on the number of ~VMs to run. Also, when multiple ~VMs were run, these were homogeneous IRIX ~VMs. Why didn't they run multiple heterogeneous IRIX and SPLASHOS ~VMs? Does SPLASHOS not afford network communication? Is there ever a use case where a SPLASHOS VM to share data with an IRIX VM?
*They disabled dynamic page migration and replication for the scalability experiment. Isn't this aspect central to the design Disco? Why not run experimental trials with and without dynamic page migration and replication to show its impact on scalability?
*Was this idea eventually implemented on FLASH? Or on loosely coupled clusters? How did it perform?
*How does the ease of implementation compare between developing a library OS like SPLASHOS for Disco and developing a library OS for an exokernel? Assume they are built for the same dedicated application.
!!!Notes:
*virtual machine monitors, multiple commodity OSes running side-by-side
*ccNUMA microprocessors, scalability, taking advantage of hardware, dedicated library OSes
*global policies, shared memory
*fault containment
*overcoming virtualization overheads (replication of large memory structures), resource mgmt (lack of application-aware knowledge to make policy decisions), sharing and intercommunication (1970s VMs looked like standalone systems)
*why now (1997) is the time (ccNUMA architectures, network protocols); NUMA memory management an impediment; solution: dynamic page migration and replication
*each node a MIPS processor, each OS given its own "physical" address space, another abstraction layer, virtualizes I/O, disks, networking, virtual CPUs
*machine page replication of virtual pages, transparent to the user, who maps to a "physical address" which is a layer above the actual machine pages
*context switches fast because of a tagged TLB, maintains physical to machine mappings, 2 levels of TLB, maintains recent translations
*dynamic page migration and replication: response to cache misses and improving locality due to NUMA non-uniform memory access times; performance benefit - even in same VM, when 2 VCPUs map the same physical page, a machine page replica is made on the node of both VCPUs
*shared copy-on-write buffer cache between all machines -> memory access pattern of a single OS, not many
*inter-VM communication via standard DFS protocols, sending of files hides transparent mapping of machine pages to physical pages of destination VM, made read-only (like IO-Lite)
*tweaking commodity OSes HAL (small changes, MIPS peculiarities and kernel address space being read-only, device drivers, network) - ported IRIX (commodity OS) and SPLASHOS (Library OC for scientific apps)
*virtual I/O and DMA
*performance eval on simulator since ccNUMA FLASH not yet ready (implications for reproducibility? SimOS developed by 3rd author) premature publication? were they afraid of being scooped? configuring SimOS to simulate FLASH (taking their word for it) - as a result, unable to answer interesting questions about long running workloads, resource sharing
*eval examined scalability, memory overhead (physical data footprint doesn't increase much as virtual machines added) and execution time overheads (up to 16% additional overhead considered modest?), memory mgmt
*how did they decide how many VMs to run? what heterogeneous/homogenous mix of VMs?
*disabled dynamic page migration and replication for scalability eval (but isn't this central to Disco?) why not run with and without to show the contribution / overhead associated with it?
*demonstrated scalability of Library OS SPLASHOS
*dynamic page migration and replication - sig. faster than comparable kernel implementation (without giving numbers, just a reference), improves memory locality, compares against IRIX and UMA implementation
*RW: Disco reduced OS development effort than scalable OSes, comparing against VMMs of the 70s, Disco has copy-on-write disks, has a global buffer cache, supports scalable shared memory multiprocessors
*comparison against microkernels and exokernels - can support multiple OSes, like library OSes, but virtualizes resources rather than multiplexes them - allowing commodity OSes to be used unmodified
*extension from multiprocessors to clusters
!!<<cite Barham2003>>: Xen
Xen, like Disco, is virtual machine monitor (or hypervisor). This paper presents the high-level design, implementation, and evaluation of a Xen system supporting a number of guest operating systems over an x86 architecture. It is worth noting that this article is the most cited SIGOPS article according to the ACM digital library, an indication of its impact over the past 10 years and its ability to clearly and convincingly convey the advantages and tradeoffs of //paravirtualization//. This approach doesn't provide a full virtualization, as in the Disco system, and instead exposes guest ~OSes to the underlying hardware to a greater extent.
Xen was designed to provide isolation and scalable performance for up to a target of 100 guest operating systems, furthermore, these would be largely unmodified commodity operating systems running potentially thousands of industry standard applications. Xen's major design features include:
*control and management via domain0, another layer of the policy/mechanism split. domain0 can arrange policy decisions, create/delete other domains for guest ~OSes, create network interfaces
*control transfer via hypercalls, asynchronous events that link applications (level 3) and guest ~OSes (level 1) to the Xen hypervisor (level 0)
*vertical movement of I/O data through the system; limiting transfer overhead and copying by pinning page frames and a producer-consumer ring abstraction.
*CPU scheduling using a borrowed virtual time algorithm, distinguishing between real and virtual time (virtual time warping); a scheduler that favours recently-woken domains.
*address translation via hypercalls to hardware page tables; batch updates and local queues of updates
*physical memory reserved at time of domain creation; providing an illusion of contiguity (as hardware memory usage may be sparse); superpages
*network I/O via a virtual firewall router and scatter-gather DMA, pinning relevant page frames until transmissions are complete, exchanging unused page frames for each packet it receives
*virtual block devices: the disk appears like a SCSI disk to any guest OS, DMA occurs between the disk and pinned memory pages in a domain, access requests are batched.
While the Disco hypervisor system is only mentioned in passing in the paper, we can compare the two systems with regards to many of Xen's design details. An obvious difference is the architectures on which they were deployed: Disco intended for use on a ccNUMA multiprocessor and Xen on an x86 architecture. This architectural difference has an important implication for the use of virtual memory by guest ~OSes. Disco made use of 2-level tagged ~TLBs, making context switching efficient. The x86 architecture required Xen to use hardware page tables; to deal with this, Xen was replicated in the top 64MB of each address space, thereby avoiding a full TLB flush upon switching contexts between a guest OS and the hypervisor. Xen differs from Disco also in respect to the extent of virtualization: Xen multiplexes resources without providing a full virtualization, choosing to omit another level of indirection between the guest OS and the hardware; this complicated the design of Disco, adding overhead and reducing performance. Because of this indirection, many of Disco memory and disk policies were transparent to guest ~OSes, such as dynamic page migration and replication, thereby inhibiting customization and optimization. Another difference relates to inter-guest communication: the Disco system provided facilities for guest ~OSes to easily communicate and share data between one another, provided by a global buffer cache and copy-on-write protocols. Xen maintained a stronger isolation of guest ~OSes; they mention an intention to provide a shared universal buffer cache as future work. This brings about a higher-level question: in practice, what are the use cases for sharing data between guest ~OSes, and how often does this happen?
At this point, comparisons can also be made at a higher level between hypervisor systems and exokernel systems, with respect to how and when they are used as well as how the two designs exhibit tradeoffs between mechanism and policy, and also between security, scalability, and performance. Both Disco and Xen support largely unmodified commodity operating systems and entirely unmodified applications. Another advantage of the virtual machine approach is that it supports process migration. Disco requires minimal changes to an operating system's hardware abstraction layer (HAL), while the cost of porting an operating system to Xen is somewhat larger, with changes beyond the HAL, requiring the changing of permission levels and the registration of exception handlers. Exokernels, on the other hand, cannot support commodity operating systems with unmodified applications; exokernels can support specialized library operating systems, each supporting a dedicated application. Thus the market for exokernels is smaller, likely for niche scientific applications that scale to take advantage of the architecture. I noticed that Disco was intended to support both commodity operating systems and dedicated library ~OSes, such as SPLASHOS. It seems that Xen's focus was entirely on commodity ~OSes, and it is unclear as to how much effort would be required to port a dedicated library OS to run over Xen.
!!!Discussion:
*Xen is a ''virtual machine monitor'', a hypervisor, supports ''commodity ~OSes'', albeit slightly modified
*goal to ''scale to 100 guest OS machines'' running industry standard applications and services
*Considering the high availability of and active development of Xen 9 years after this paper was published, it's safe to say that Xen itself was a relatively major contribution
*Most cited SIGOPS paper
!!!!Arguments for virtualization
*guest OS isolation/security, protection of resources, good for scalability, no need to configure complex interactions between supposedly disjoint applications; in single OS systems, resources may be oversubscribed, users uncooperative/untrusting, process migration, admission control / accounting for resources
!!!!Arguments against full virtualization
*complexity, reduced performance, high overhead cost for update-intensive operations, creating new applications, lack of transparency: guest ~OSes don't see physical physical resources, only virtual mirror
!!!!VM use cases:
*Xen very similar to Disco, but in spite of achieving similiar ends, both the motivation and mechanisms used are very different
**''Q'': when to use paravirtualization, full virtualization, exokernel approach
**''Q'': Does usage of virtual machine systems reflect the Disco model or then Xen model? How big is the market for virtual machine monitors in the domain of scientific applications (e.g. SPLASHOS, exokernel library ~OSes), relative to the use of virtual machines for accessing web services?
**''Q'': Can you virtualize the virtualizer? assume not w/ approach of Xen/Disco, b/c of how trapping is streamlined
**''Q'': sharing between guest ~OSes Disco supported this but Xen did not (listed as FW)
*''future/related work'': passing reference to Disco; wants sharing b/w guest ~OSes, copy-on-write semantics, not sacrificing isolation, implications for accounting, auditing, forensic logging
!!!!Design principles
*Xen multiplexes resources, not full virtualization: ''paravirtualization'', requires OS modification, but no modifications to guest applications, exposes machine addresses, guest ~OSes see real and virtual addresses: "similar but not identical to the underlying hardware"
*''separation of mechanism and policy'', each guest OS controls how it uses memory and disk; security and visibility of physical resources
!!!!Architecture & Memory management
*Xen built over the ''x86 architecture'', which uses ''hardware page tables'', not a software-managed TLB, more difficult to manage memory
*Xen has to use meta instructions from guest operating systems to manage the page table between all of them; batch updates from guest ~OSes to page tables, write-only page tables
**''Q'': Disco's ~ccNNUMA vs. Xen's x86 UMA?
**''Q'': ''instructions were added to x86 to aid in virtualization''. I imagine ~VMWare has now closed the preformance gap substantially because of this, what is the nature of these instructions?
**''Q'': x86 has ''instructions to support virtualization'', and Xen can run guest ~OSes with full virtualization in this way. How quickly did the virtualization instructions come out after Xen? full virtualization for running unmodified guest ~OSes does not perform as well as paravirtualization with modified ~OSes. Is tradeoff significant enough today to continue the development of guest ~OSes tailored to Xen? Would Xen have had as much success if it had been released after the virtualization instructions were introduced into x86?
*''memory management'': Xen in top 64MB of every guest OS address space, so no complete flushes are required when entering or leaving the hypervisor
*the top 64MB may not be accessible or mappable by guest ~OSes, to allow Xen to reside there to avoid TLB flushes during crossings
**''Q'': Why did most operating systems leave the ''top 64 MB free in the address space'', this seems awfully convenient?
**''Q'': How does having Xen in 64MB at the top of every guest OS's adddress space avoid TLB flushes when entering/leaving the hypervisor?
*guest ~OSes have segmented address space
!!!!The CPU
*guest ~OSes operate at ''lower privilege level'', uses 3 rings of protection, requires little modification to guest ~OSes; requires registering exception handling protocols,
*xen virtualizes exceptions: system calls and page faults -> hypervisor calls (''hypercalls''); double faults terminate guest OS
*operations like forking a process suffer from performance degradation because of the number of hypercalls they must make. However, this problem is somewhat mitigated because guest ~OSes are allowed read-only access to the page table without interference from Xen. Only updating the page table requires a hypercall to Xen
**''Q'': What good were Ring 1 and 2 for originally, or more specifically what can you do in them, but not in ring 3, and what does ring 0 allow you to do that ring 1 doesn't.
**''Q'': The requirement of every ''privileged instructions to be validated by Xen'' still seems to me to be a performance overhead.
**''Q'': confusion re: terminating guest ~OSes if they cause a ''double fault''?
**''Q'': Using an exokernel like approach, the operating systems no longer directly execute privileged instructors in the kernel, (in Disco they would be trapped, emulated by the hypervisor, and then control return), in this system they use a hypervisor meta instruction to complete the request.
!!!!Porting commodity ~OSes
*''ease of implementation'': Xen requires little modification to Linux, somewhat more to XP
**''Q'': ''Argument for full virtualization'', a virtual machine should be fully virtualised, impossible to distinguish from real hardware. drawbacks of needing existing ~OSes to be modified means that it does not provide as much isolation or compatibility as desired
**''Q'': operating system ''modifications may not be easy'': difficult to modify Windows XP in a way to operate with Xen. modifying OS code would make it increasingly challenging to achieve generality of Xen
**''Q'': ''Changing an OS'' to make it run on Xen ''NOT trivial''! 3000 lines of Linux actually requires a programmer to be skilled with both the Linux kernel and with the PV interface exported by Xen. Not to mention this must be done for every OS kernel that needs to be run on top of Xen.
**''Q'': will overhead be similar for the ~WindowsXP and ~NetBSD ports?
**''Q'': XP port is incomplete, it was never published / demonstrated? never finished?
**''Q'': changing Linux source is ''not'' at all ''feasible'' unless Xen reaches critical mass and becomes big enough that every major OS developer will include this Xen stuff as part of the kernel. Luckily, that's exactly what happened as Xen's performance gains from PV were totally worth it.
**''Q'': the ''distinction b/w OS and application'' isn't clear cut, is it really possible in the general case to avoid modification of applications for Xen, ''if the ABI and OS are sufficiently close and interdependent''?
!!!!Control and management
*via domain0 (another layer of policy/mechanism split), domain0 can arrange policy decisions, creating/deleting other domains, network interfaces, administrator, builds new domains
!!!!Detailed Implementation
!!!!!Control transfer
*hypercalls (synchronous software traps for downcalling) and events (asynchronous upcalls, sparingly used), callback handler
**''Q'': interrupts replaced in domain operating systems with a series of event queues that act very much like interrupts, but have additionally semantics to support signalling from the virtual environment
!!!!!I/O
*vertical movement of i/o data through the system, little overhead and copying - pinning page frames; producer-consumer rings
**''Q'': ''I/O descriptor ring mechanism'': claim that use of descriptors w/ out-of-band buffers make implementing zero-copy transfer easy
**''Q'': I/O and network implementation as described reminded me of ''~IOLite''; are there some distinguishing differences that I have overlooked?
**''Q'': the I/O producer-consumer protocol
!!!!!CPU scheduling
*borrowed virtual time, real vs. virtual time, virtual time warping, scheduler favours recently-woken domains
!!!!!Virtual address translation
*no shadow page tables, hardware page tables, hypercalls, allocation and use of page frames, registering pages tables with guest OS, type safety of pages, maintaining reference counts, batch updates, locally queue updates
*Xen mediates every access, avoiding the need to keep shadow page tables
!!!!!Physical memory
*reserved at time of domain creation, more available upon request, illusion of contiguity, balloon driver, hardware memory usage may be sparse, superpages
**''Q'': domains are given a ''static amount of memory'', with the opportunity to ''request more up to a predefined limit''. Why was this strategy chosen? Does the restriction of remaining "similar" to the underlying hardware prohibit strategies such as ''visible revocation''? setting a hard limit on the memory consumption of an OS is ''inflexible''.
!!!!!Network
*virtual firewall router, scatter-gather DMA, pinning relevant page frames until transmission complete, exchanges unused page frame for each packet it receives
!!!!!Disk
*appears like a SCSI disk to any guest OS, DMA b/w disk and pinned memory pages in a domain, virtual block devices, batch requests
!!!!Evaluation
*comparison with ~VMWare workstation (not ESX server), and UML (user mode linux); effort to level playing ground, disabling hyperthreading
*relative performance: benchmarks: SPEC ~INT2000, linux build time, ~OSDB-IR, ~OSDB-OLTP, dbench, SPEC ~WEB99; places differing loads on ~OSes; performance in Fig 3; microbenchmarks in tables 3,4,5, network benchmarks in table 6
*concurrent ~VMs: 1,2,4,8,16 machines, doesn't get worse with more ~VMs on SPEC ~WEB99 benchmark
*''performance isolation'': simulated multiple ~OSes along with antisocial processes, better than on native Linux
*''scalability'': small memory footprint per guest OS (20kb?), no shadow page tables
**''Q'': Can you really put a clause in an ''EULA'' that prevents you from publishing benchmark results?
**''Q'': Is it fair to restrict someone from reporting the benchmark times for commercial software? Doesn衴 stand in the way of scientific progress? It seems unreasonable.
**''Q'': The lack of benchmark results due to license agreements is disappointing. On the other hand, ''~VMware has published a performance comparison'' that shows, not surprisingly, ESX beating Xen by a large margin. Apparently ~VMware only allows publishing benchmark results that show them performing better: should this be allowed? Other independent benchmarks showing ''Xen and ESX have approximately the same performance''. It's worth noting that ESX is a full virtualiser, unlike Xen which is paravirtualising.
*''Q'': Most micro-benchmarks show a bigger performance gap, but they note some spurious results and seem to suggest that micro-benchmarks aren't particularly useful.
*How does User Mode Linux's approach work (probably very unrelated).
!!!!notes from Clark et al //Xen and the art of repeated research//:
*Clark et al (//Xen and the art of repeated research//, proc. USENIX '04) revealed that the scalability evaluation of Barham et al in 堯nly allocated 15MB for each guest, pointing out that this is not realistic nor sufficient for an industry standard web server. A 128MB memory size per guest would be more typical, however this would require over 12GB of memory. They state that an upper bound of 16 guests is more realistic for a system with 4GB of memory.
**Linux kernel supports 15 partitions. [Xen03] patched the kernel to allow 128 raw disk partitions.
**They flip-flopped between running Linux with and without SMP support.
!!<<cite Klein2009>>: seL4
seL4 is a recent addition to the L4 microkernel family, one that was formally verified for functional correctness throughout its design and implementation, ensuring an unprecedented level of reliability and predictability. This article describes the authors' arduous efforts to prove the correct behaviour of the system at several levels of abstraction. In effect, they have engineered a microkernel system that exhibits precisely specified desirable behaviour, bug-free, and (apparently) comparable in performance to related L4 systems. As in other microkernel systems, there is a strong separation of mechanism and policy, the latter being relegated to user-level processes. The formal verification effort is therefore concentrated on the correct workings of mechanisms defined in the kernel, providing no guarantees that user-level policy make sense. Despite the leanness of microkernels, the proof and implementation effort they describe is substantial, requiring upwards of an estimated 20 person years (!) to complete.
The contribution of this paper is a design and implementation methodology, one that offers a compromise between top-down formal verified software specification and traditional bottom-up kernel design: a staged refinement from an abstract design to a concrete design via a mid-level evaluation of a deterministic Haskell prototype implementation. The purpose of this mid-level stage is to catch implementation bugs before fully specifying the system in a pragmatic subset of C. The functional correctness proofs tend to be highly interrelated, so changes at the concrete implementation level tend to affect large areas of the proof space, requiring a substantial reverification effort, as reported in their section on "lessons learned".
The proofs themselves decompose along function boundaries, detailing preconditions, statements that modify system state, and post-conditions. The specification of these proofs is a trade-off between minimizing complexity and ensuring performance. Among the challenges faced when specifying these proofs were global variables and their side effects, as well as issues relating to concurrency. In the former case, fully specifying side effects in the proof brings their attention to the design team. They solve (or sidestep) the latter issues by implementing all I/O as user-level processes with an event-based polling approach with explicit interrupt points. Issues with synchronous exceptions are also avoided by mapping the kernel code to a fixed region in every virtual address space, much like in the Xen hypervisor system. The bulk of their proof process was dominated by four types of invariants: low-level memory invariants (relating to restrictions on object size, placement, and overlap in memory), typing invariants (objects of well defined and context dependent type), data structure invariants, and algorithmic invariants (these being the most complex, which often ran the risk of violating other invariants). Altogether, the proofs ensure that all kernel behaviour is always defined and will never crash.
!!!Discussion:
*There was no full or convincing performance evaluation, though they state that performance was comparable to that of related microkernel systems. Was the effort worth it? It is assumed that there are use cases where one would opt for a fully verified microkernel (an embedded system?). Though I assume the price point would be much higher than that of an unverified microkernel.
*Regarding the separation of mechanism and policy: only the mechanism is verified, and no matter what user/application-level policy is given, behaviour of the kernel can be predicted. They do not guarantee that their specification describes the behaviour the user expects. Is the problem such that users can't articulate what they expect, or is it that users differ in terms of what they expect? Can a user model be formally specified?
*Much of the related work cited describes systems that were verified but never implemented, implemented but never verified, left many parts of the kernel unspecified, proved data separation but not functional correctness, or were not sufficiently complex enough for a general purpose microkernel. Is the overarching problem a lack of human resources (time, funding)? Does this work succeed simply because they had a team committed to dedicating 20+ person years and a large body of funding (likely less than $87M)?
*Their verification was not complete, as it rests on assumptions that the compiler assembly code, and hardware are reliable. If the seL4 system is to be portable to other architectures, are these assumptions overly strong?
!!!Notes:
*Microkernel of the L4 family, resulting performance comparable to other L4 variants
*formal verification guarantees absence of bugs, functional correctness, behaviour fully specified, desirable behavior can be specified at several levels of abstraction, trustworthy, confident
*formal verification process as described is a methodology for kernel design and implementation: formal verification is top-down, while most kernel design is bottom-up; their method is a compromise; a correspondence or refinement from abstract to concrete
*separation of mechanism and policy; the latter is user-level and need not make sense; what matters is the mechanisms that they rest on; memory management mechanisms an instance of this (⩊*they implement and evaluate the abstract and kernel level specification using a Haskell prototype, then reimplemented in an optimized subset of C
*lessons learned: IPC performance most critical; effort considerable for a microkernel (proof along took 20 person years, likely $87M at $10k/line of code; many small defects found during implementation stage; late changes made to the kernel can incur long periods of re-proving (esp. large, cross-cutting features - 1.5 -2 person years to reverify in some cases)
**over 150 invariants over the different levels, many interrelated
*global variables and side effects make verification hard, pointers and temporary violation of invariants (e.g. linked lists in scheduler algorithms); side effects need to be made explicit
*concurrency proofs are hard (uniprocessor only), async I/O interrupts and events; they avoid these complexities by implementing these at user-level with an event-based approach using polling, interrupts are mostly disabled, using interrupt points instead; synchronous exceptions guaranteed to never to produce fault, since kernel code is mapped to fixed region of virtual address space
**abstract level: specifies //what// but not //how//; finite state words, multiple correct results for an operation are all specified; no scheduling
**executable level: specifies //how//, reflects restrictions in size and code structure of C, deterministic
**C implementation: most detailed level, assumptions about lower levels (compiler, architecture), a large, pragmatic subset of C (limits side effects in expressions, no function calls through function pointers / reference parameters, no unions)
**lower level machine model in small assembly code base, accessible via interface
**the proof: correspondence/refinement of transitions b/w abstract and implemented states: kernel transitions, user transitions, user events, idle transitions, idle events; does not guarantee that the specification described the behaviour the user expects; nevertheless, behaviour is always defined, all kernel API calls terminate and return to user level; invariant proofs dominate the proof: low-level memory invariants (object size, placement, overlap restrictions), typing invariants (well defined, context dependent type pointing), data structure invariants, algorithmic invariants (most complex, must ensure doesn't violate other invariants)
*RW: automatic techniques: static analysis, model checking, shape analysis does not capture full functional correctness; type safety not strong enough;
*overview: seL4 has untyped capabilities (unlike other L4 systems); device drivers are user-level applications, handled by IPC;
*verification not complete, rests on assumptions of compiler, hardware
*I/O device drivers are user-mode components, except timer driver
*verification:
*FW to verify assembly and application levels (verification stack of verisign)
!!<<cite Swift2003>>: Nooks
Many of the failures that occur in commodity operating systems are caused by kernel extensions, such as: device drivers, optional file systems and other application-specific extensions. OS extensions are increasingly prevalent, and the cost of failure continues to rise. This paper describes a technique for ensuring system reliability by isolating and recovering from these failures, implemented as a kernel subsystem called //Nooks//. Nooks does not require modifications to existing commodity OS kernels or extensions, ensuring backward compatibility with existing extensions and operating systems. Instead, the Nooks subsystem is a transparent intermediate layer between extension code and kernel code, wrapping extensions and channeling communication from kernel to extension via extension procedure calls (~XPCs). This layer contains the Nooks Isolation Manager, which handles isolation, interposition, object tracking, and recovery from faults emanating in potentially buggy extensions. This boost in reliability does sacrifice performance to varying extents, dependent on the type and behaviour of extension, which is demonstrated later in the paper.
The key components of the Nooks Isolation Manager include:
*''isolation'': extensions are separated into their own lightweight kernel protection domain, communicating with the kernel via ~XPCs. They note that this is similar to the management of address space in single-address operating systems (such as Opal). Like the designers of Xen, they faced the same x86 architectural issue associated with context switching and expensive TLB flushes. This is avoided by having extensions share part of the kernel address space. Also like hypervisor calls and events in Xen, ~XPCs can be deferred and batched to reduce overhead.
*''interposition'': function pointers to and from extensions are are now routed through wrappers, which appear to extensions as part of the kernel's API. From the kernel perspective, they appear to be extension entry points. These wrappers are in the kernel's protection domain, where data references can be checked or copied as necessary. Furthermore, these wrappers can often be shared by several similar extensions.
*''object tracking'': objects passed between kernel and extension are recorded with unique types and identifiers, labelled as being objects for single use or objects for repeated use. This component also performs garbage collection, cleaning up when an extension fails.
*''recovery'': a recovery manager and user-mode recovery agent suspends faulted extensions, releases their resources, and restarts the extension.
The authors demonstrate how Nooks increases system reliability in an experiment where faults were injected into several types of kernel extensions (device drivers, an optional file system, a web server extension). These faults would normally result in fatal system crashes; Nooks isolates and recovers from 99% of them, reducing data corruption cases from 90% to 10%. Nooks can also recover from a sizeable number of non-fatal extension failures (a 60% improvement), particularly exceptions that emanate from user processes, which tend to elude Linux. Unfortunately, performance drops as a result of adding the Nooks layer, ranging from a 10% to 60% slowdown for some applications, compare to those running on an unmodified Linux system. The authors suggest that the reliability-performance tradeoff should be considered on a case-by-case basis.
It's worth comparing Nooks to other systems that we have discussed. Systems using type-safe languages, such as Pilot, may be reliable, however they cannot run commodity operating systems, nor do developers write extensions for them. Capability- and segment-based architectures can also improve reliability, but again these do not often support the existing code base of commodity and legacy operating systems. The same could be said of microkernels. Fault isolation is a key design principle of virtual machine monitors such as Xen, and while these support commodity operating systems, they are intended for a specific use case, one in which virtual machines do not share between one another. Nooks is targeted at the general use of commodity ~OSes, one which leverages the benefits of sharing between applications an operating system.
!!!Discussion:
*Because modifications to the hardware, OS, and extensions are not required, is Nooks's installation and configuration process straightforward? Could the average user do this? Who is the target user?
*We have to take them at their word that synthetic fault injection was an externally valid test of the system, that these faults are representative of possible day-to-day usage. The faults may be biased toward the extensions selected in the experiment. Admittedly, it would be hard to study the reliability of Nooks, short of collecting data from a longitudinal A/B field study.
*Not much was said about the usage of the application-level Nooks recovery agent. How does the user interact and configure this service?
*A configurable dashboard with bypass options could allow users to specify when the Nooks layer should be bypassed, thereby explicitly controlling the performance-reliability tradeoff. The authors admit that some ill-performing extensions (such as the web server) may represent a poor application of Nooks. A flexible compromise would allow users to run without Nooks except when explicitly requested.
*Some extensions share wrappers. Does extension-to-extension communication also occur via the Nooks wrapper and XPC? Or can extensions call each other directly with no communication overhead? Do use cases exist for this?
!!<<cite Efstathopoulos2005>>: Asbestos
Existing operating system security policies aren't fine-grained or flexible enough for military-grade usage, where it's really important to prevent any user's information from being leaked. So DARPA funded this research to build a new �cure operating system, of one which incorporates and extends the concept of capabilities. The Asbestos OS uses a kernel mechanism called a label, which applications can use to dictate a range of finely-tuned policies, which in turn give application designers flexible control over access control and the flow of information between processes and other subsystems.
From what I understand, labels have the combined functionality of capabilities and access control lists, in that security is discretionary at the point of entry and based on the entity (process) requesting entry. The way the authors describe it, there are send, receive, verification, and port labels, and these can interact in a number of ways based on their permitted contamination level. These label interactions include means to transitively isolate processes from one another and thereby dynamically adjust labels, compartmentalizing information without any central control (which I suppose would be hard to do with either basic capabilities or access control lists). Compartmentalized information is referred to by a kernel data structure called a handle, which exhibits a level of contamination; a label is a function of a handle's contamination level (I think). The handles don't convey any user-visible information, which makes the handles difficult to exploit. The logic of label interaction is based on the principle of least privilege, which recalls Saltzer's 1974 Multics paper, which presented similar use cases of users with shifting "need-to-know"-level security clearances.
The other key component to the Asbestos system is the //event process//. These complement labels in that labels are associated with compartments that may be shared between users, while an event process abstracts a single user's state. Event processes are a better alternative to user-level threads, which do not provide isolation. Event processes make use of a few new system calls that allow for isolation without the usual system call overhead, as this expense would build up when forking for each new processes for each user process. It turns out that using event processes is really space efficient (~1.5 pages of memory per user in their web server evaluation). On the other hand, event processes limit concurrency, which accounts for poor throughput performance. The authors spin this in a positive light, since concurrent malicious processes may be able to use covert channels to exploit an information channel; if you limit concurrency, you can limit the use of covert channels.
The application they had in mind while developing Asbestos was a secure dynamic-content web server, where it is imperative that multiple connecting users are isolated from one another. Their server required dynamic security policies making flexible use of the label mechanism. Their implementation turned out not to perform very well when scaling up to load levels served by Apache, but the authors seemed pretty confident that the system was so secure that the performance hit would be worth it.
//I'm imagining that a condition of DARPA's funding for the Asbestos project was that any published paper emanating from research would be required to present it's technical content using the most staggeringly incomprehensible prose, so as to preclude the reproducibility of the work, or to obfuscate some vulnerability. There's probably a classified ~DARPA-internal tech report that spells it all out in a straightforward style䠳eriously, folks, how long does it take for a mere mortal to wrap their head around section 5? I'm still mystified. And I've spent far too many hours on this paper alreadyᡡDiscussion:
*Backwards compatibility wasn't a priority for these guys. I'm guessing that it would require a significant porting effort to make legacy applications use these new system calls: what effort was required to retrofit their standard database? Instead, their goal was to proactively construct a secure platform. So did anyone write applications for Asbestos? The Asbestos site hasn't been updated for 4 years, so I'm assuming that the OS is no longer under development. A quick literature search on citing articles tells me that the label mechanism is the lasting contribution of this paper, as it appears that labels as general capabilities spread to the design of other recent systems (~HiStar, Flume).
*are //taint// and //contamination// synonymous?
*"psychologically, however, people have not accepted pure capability-based confinement" (꠰eople think access control lists are easier concepts to grasp? I expect that the complexity of labels poses its own psychological angst.
!!<<cite Yip2009>>: Resin
Resin is a system that ensures the security of data passing in and out of applications. The intended usage of Resin is similar to that of labels and event processes in the Asbestos OS, which afforded application developers a flexible degree of control over the flow of data through an application, thereby securing a system from malicious adversaries. Both assume a trusted application code base, and both tradeoff performance for increased security. However, Resin differs in that it is not an operating system in itself, but a language-level runtime which integrates easily with existing applications; it presents a convenient PHP and Python API, and can reuse existing application code, in effect allowing the application developer to make assertions about data flow explicit. It also differs from Asbestos in its simplicity: while the Asbestos OS relied on a complicated and dynamic logic to reason about information taint, Resin provides a sufficient amount of application security with three lightweight concepts: //policy objects, filter objects//, and //data tracking//. Even a proficient, well-intentioned developer may occasionally overlook a vulnerable access point in an application; such programming errors are inevitable in large applications. If used in conjunction, policy objects, filter objects, and data tracking ensure that these vulnerabilities cannot be exploited in common ways such as SQL injection, script injection, and information leakage.
The authors use common security vulnerabilities as illustrating examples throughout the paper, which I appreciated; these use cases allowed for an easier understanding of the implementation and usage of Resin and its three main components. This approach is in stark contrast to the approach taken by the Asbestos authors, who opted not to ground their explanation in tangible use cases. At a high level, the design of Resin could be summarized as follows:
*''Policy objects'' are language-level annotations on datum, which dictate how that datum can be exported from the application.
*''Filter objects'' are language-level annotations on channels of data flow, which examine the export checks of policy objects of data flowing through the channel. They are placed at I/O boundaries to the application, including a default boundary, which covers corner cases overlooked by developers.
*''Data tracking'': Policies are propagated when data is copied, and policies are stored and retrieved whenever data is saved to or requested from disk. A datum can have multiple policy objects associated with it, and since policy objects can be specific to individual characters, developers are given a choice as to how policy objects are merged when data is concatenated; concatenated data can either inherit a union or intersection of policies. When data is split, component data will not inadvertently retain policy objects not originally held prior to a merge.
Resin is evaluated in several capacities. In terms of programmer effort, Resin is very attractive, as existing code is reused to express assertions in only a handful of LOC; the application design and structure remains unchanged. When they evaluated Resin in terms of its ability to detect vulnerabilities, they found expected as well as several unexpected vulnerabilities, both specific and general. However, the methodology of this evaluation was not clearly stated, as the results are conveyed anecdotally. Lastly, they present (and downplay) the considerable performance overhead incurred by integrating Resin with applications. Storing/serializing and retrieving/deserializing policy objects upon SQL select and insert operations are particularly slow. Their section on limitations and future work indicates that optimizations leading to performance improvements are forthcoming.
!!!Discussion:
*Despite the performance overhead, Resin provides a sufficient amount of confidence that application data flow is secure; in other words, the security-performance tradeoff is still not so severe. On the other hand, their solution integrates with applications easily and is easy to reason about (as opposed to Asbestos); is this why we can ignore the poor performance?
*In the Nooks paper, we learned that OS extensions contain are where most of the OS failure points exist; here, we learn that many application-level vulnerabilities exist due to third-party programmers. Nooks and Resin are similar in many ways: interposition, object/data tracking, isolating entry points. Nooks provided a recovery agent to provide system status and to give the user interactive control over system recovery; could Resin provide something similar, a real-time indication of attempted security breaches?
!!<<cite Ghemawat2003>>: Google FS
In this article, Ghemawat and colleagues present the design and evaluation of the Google File System (GFS). The architecture and policies of the GFS are unique to Google's application workloads, wherein the assumptions that guide traditional file system design no longer hold. The GFS has its own set of assumptions: high-bandwidth interdependencies with Google applications, multiple clients, numerous component failures, a common case of large files, a common case of large reads, and file mutations where appending is more common than overwriting.
The GFS is summarized up by Figures 1 and 2, and much of the paper describes the architecture and operations between GFS's components. Files are stored in chunks, replicated several times on physically distributed //chunkservers//, each maintaining metadata for the chunks, including a version number. Meanwhile, a //master// polls the chunkservers, maintaining an awareness of which chunkservers are running, which chunks they contain, and where replicas are stored; it occasionally relocates and re-replicates chunks according to dynamic circumstances, maximizing reliability and availability across distributed chunkserver locations. It also contains an operation log, which can be replayed in situations where the master fails. Finally, multiple clients will ask the master for the location of file chunks; the client subsequently reads and writes to the chunks directly, taking advantage of pipelining and the network topology whenever possible. Chunks are allocated lazily by the master, and are of a fixed size. Similarly, garbage and stale replica collection is performed lazily after a configurable delay interval. Much of what follows is dedicated to how writing to chunks occurs. Novel file system operators //snapshot// (checkpointing) and //record append// serve to resolve consistency issues during write operations across primary and replica chunks.
As components of the GFS are highly replicated (including the master), the system is highly available and reliable, recovering quickly from component failures. Checksumming maintains data integrity and consistency, and can be used to diagnose potential failures as well as concurrent writes from multiple clients. The system performance is lower than expected, despite their design choice to limit a communication bottleneck at the master: directing data flow between clients and chunkservers, rather than via the master. I imagine that the master is not the source of the performance problems; it is more likely caused by data flow and consistency checks between chunkservers. Finally, they evaluate GFS both with microbenchmarks and with real world clusters, and it was interesting to compare clusters that meet GFS's assumptions (Y) with those that don't (X), where it was evident that the workload tended to be dominated by appending rather than overwriting.
!!!Discussion:
*Are there specific Google applications that are likely candidates for the types of applications described in this paper (such as those in Clusters A,B,X,Y); which of these application tend to favour overwriting and which tend to favour appending?
*I was confused by the section describing the consistency model (穻 perhaps a flow chart could have described the arrangement between a //(un)defined// file region and a //consistent// file region; it was difficult for me to keep track of consistency issues between the primary and replicas, while at the same time having to reason about concurrent writes from multiple clients.
*I question the external validity of their append microbenchmark (᮳)? Wouldn't a representative system have N clients appending to M files, and not to a single file, which is more in line with what occurs in practice.
*Why was the append rate have relatively higher variance with more clients (Fig. 3c) than that of the write rate (Fig. 3b)? If GFS was optimized for the append operation, I would expect the reverse. (I admit that this observation may be an artefact of differing y-axis scales).
!!References
<<bibliography Bibliography-CPSC508 showAll>>
!!Performance Evaluation in Operating Systems Research: Approaches and Challenges
!!!Abstract
A survey of four research papers pertaining to performance evaluation of operating system is presented. This survey and its related discussion highlight the approaches to system evaluation along with its associated challenges and tradeoffs. The article speaks to methodological issues of realism, accuracy, the granularity of measurement, the portability of measurement techniques and tools, as well as the comparability and reproducibility of methods and results.
!!!Introduction
It has been observed that in operating systems research, most of the intellectual effort goes into the implementation of novel systems (<<cite Mogul1999 bibliography:Bibliography-CPSC508-Project>>). This is not surprising, as researchers must understand the complex multifaceted relationships and design tradeoffs between speed, security, extensibility, scalability, portability, and complexity. However, it has been argued that more effort is needed when it comes to evaluating the efficacy of these novel systems (<<cite Seltzer1999>> ,<<cite Small1997>>,<<cite Traeger2008>>), and that performance evaluation protocols and benchmark workloads often lead researchers to "measure what is easily measured" (<<cite Mogul1999>>). A fair and valid evaluation of system performance requires its own set of tradeoffs and challenges: representativeness of real-world behaviour, accuracy and the granularity of measurement, the portability of methods and measurement tools, the comparability of results between systems, as well as the reproducibility of methods and results. Furthermore, the protocols and results of operating system performance evaluations are seldom reported in a consistent or standardized fashion, relative to other areas of computer science (<<cite Mogul1999>>, <<cite Small1997>>), particularly in research pertaining to file systems (<<cite Traeger2008>>).
Computer science is still a nascent discipline, and there are signs of emerging standardization with regards to how computer hardware and software are evaluated. As the execution and reporting of evaluation experiments becomes increasingly transparent and reproducible, and as methodologies become increasingly transferable, it is believed that this will lead to higher rates of technology transfer from research to industry, to the successful deployment and adoption of novel operating systems (<<cite Clark2004>>, <<cite Mogul1999>>).
This paper describes the approaches and challenges associated with the empirical evaluation of operating systems and their components. It was motivated by my own research interest in experimental methodologies, owing to my background in human factors and the cognitive sciences; while I am familiar with methods for studying human behaviour and human-computer interaction, I was unfamiliar with how system behaviour is studied prior to performing this survey. My questions related to how system performance is measured, which metrics matter, and how procedures and results ought to be reported.
To facilitate a review and comparison of research papers pertaining to operating systems performance evaluation, I began my survey with several relevant position and workshop papers (<<cite Bershad1992>>, <<cite Liedtke1998>>, <<cite Mogul1999>>, <<cite Seltzer1999>>). My intent was to complement my own opinions on the need for evaluation, on experimental design and reporting, as well as on issues of experimental reproducibility and validity. These position papers also directed my literature search, leading me to three of the articles (<<cite Brown1997>>, <<cite Chen1996>>, <<cite Traeger2008>>) summarized in the survey presented in the survey section.
The structure of this paper is as follows: in the following section, I describe the several dominant approaches to performance evaluation in systems research, indicating when said approaches are appropriate, highlighting their advantages and disadvantages. These articles speak to several methodological issues relating to performance evaluation in systems research, namely the challenges and tradeoffs mentioned above. Following this, I discuss common themes and draw comparisons between the articles in the discussion section. Finally, I suggest areas for future work along with my conclusions.
!!!Performance Evaluation: Approaches
<<cite Mogul1999>> reviewed the various approaches to measuring and reporting operating system performance, and observed a tension between realism and reproducibility.
!!!!Benchmark Performance
A common approach involves the use of //benchmark// protocols or workloads, often these are the product of prior academic research, or they are made available by technology vendors. These benchmarks are often easy to use and provide simple numeric results. <<cite Liedtke1998>> elaborates on the uses of benchmarks: while benchmarks serve a useful commercial purpose in communicating system capabilities to customers, benchmarks are used in research to understand a complex system, to characterize or predict the effects of a modification, to help identify performance bottlenecks, to invalidate a theory, or to formulate a hypothesis.
Preexisting benchmarks are often inflexible in that they tend to lack configurable workload sizes or runtimes (<<cite Mogul1999>>); in addition, <<cite Liedtke1998>> describes several other problems associated with the use of benchmarks: One of which is //non-transitivity//, in that combining multiple optimizations proven separately by benchmarks may not result in a positive increase in overall system performance, due to unforeseen interactions between the optimizations. Another problem is //instability//, where low-level optimizations have a positive effect on benchmark performance but not on application performance, even when benchmarks are designed with target applications in mind. Finally, there is the problem of //non-linearity//, where different system behaviour and performance may result from alternative concurrent and sequential benchmark runs.
Both <<cite Mogul1999>> and <<cite Liedtke1998>> discern between //microbenchmarks// and //macrobenchmarks//. Microbenchmarks are useful for measuring the performance of a system primitive in isolation, but a microbenchmark in itself cannot by itself be used to predict the performance of the whole system. <<cite Traeger2008>> suggest that microbenchmarks and macrobenchmarks be used in conjunction as a means of evaluating a system in its entirety. <<cite Seltzer1999>> argue that microbenchmarks which do not consider application usage patterns and representative loads on these primitives are of little use to fellow researchers or potential adopters of the system.
When microbenchmarks are used to evaluate low-level system primitives, <<cite Bershad1992>> highlight some of the additional assumptions that often contribute to misleading results of microbenchmark experiments. Microbenchmarks often do not account for cache and buffer effects incurred by real applications: performance can vary from run to run of an application, or between different versions of the same program when cache effects occur. As a result, microbenchmarks tend to overestimate the use of some system primitives, using them in ways that wouldn't be reflected in practice. To make microbenchmark results more reproducible, Bershad et al. suggest that the cache be explicitly flushed between trial runs, thereby revealing worst-case, albeit reproducible, performance.
Macrobenchmarks are intended to measure and predict the higher-level end-to-end performance, simulating application workloads by performing a mixture of operations, while maintaining the analyzability that a benchmarking tool affords. Since macrobenchmark measurements are distinct from directly measuring the performance of any particular applications, the representativeness of a macrobenchmark workload is debatable.
!!!!Application Performance
Another common approach to performance evaluation is what <<cite Mogul1999>> refers to as "running a bunch of programs". While a seemingly ad hoc selection of applications raises doubt about whether they are indicative and representative of general system performance, this approach does provide a realistic indicator of performance if the system under measurement was specifically designed to support these particular applications. On the other hand, application-specific benchmarks are often not reproducible, and it is difficult to make meaningful comparisons across systems or over periods of time.
<<cite Mogul1999>> calls upon researchers to design benchmarks that are more realistic and representative of real-world applications, while maintaining the ability to measure and predict absolute performance in production environments. This sentiment is shared by <<cite Seltzer1999>>, who proposed several methodologies for application-specific benchmarking. One of which is a //vector-based methodology//, in which a characterization of a system in terms of its underlying system primitives is represented as a vector. Meanwhile, a second application vector describes the demand the application places on each primitive. The combined dot product of these vectors is reported as a measure of system performance. The advantage of this methodology is that it generalizes whenever applications are implemented using a common well-defined operating system API. A disadvantage pertains to the granularity of the results; relative application performance is often correctly measured, but absolute performance measures across system and application configurations is more difficult to attain.
!!!!~Trace-Based Approaches
Tracing system behaviour is an approach to system evaluation that involves recording and replaying recorded real-world workloads. Tracing can provide highly realistic performance data, but since changing the system implementation often changes user behaviour, <<cite Liedtke1998>> warned that the recorded trace may no longer be representative, and any evaluation of the changed system that uses the trace may lack external validity. When replaying a trace, it is not clear when to trigger recorded actions in response to system behaviour, because one cannot account for how a user's behaviour changes in response to novel system behaviour. In addition, studying user interaction is a problem for reproducibility, even when no system changes have been made, since user behaviour is inherently irreproducible and unpredictable.
Nevertheless, replaying traces on modified systems can still be informative. If one is confident in the representativeness of a trace despite a change in system behaviour, confident that any effects on user behaviour will be minimal, the trace could simulate realistic load for comparing different policy implementations, such as those for process scheduling or page allocation. <<cite Mogul1999>> observed that trace-based evaluations are also prevalent in mobile and embedded systems research, where controlled laboratory settings are not representative of the context in which these systems will be deployed.
<<cite Seltzer1999>> described a //trace-based methodology// for constructing application-specific benchmarks, in which usage loads are modelled after particular streams of application use. These trace-based benchmarks make it is easy to construct what-if scenarios and mimic anticipated application workloads. Additionally, these trace-based benchmarks can be disseminated and shared more easily than application logs, which are often proprietary or known to contain private information. However, these traced-based benchmarks are specific to an application-system combination, and do not give insight into the use of underlying system primitives seen in the aforementioned vector-based methodology.
When designing a trace-based evaluation, one must be concerned with the granularity of the trace (<<cite Liedtke1998>>); traced events should be large enough to label with semantic meaning, without being intrusive enough to have the user notice the overheard incurred by the trace. Related to the issue of trace granularity is whether there is enough information contained in a trace to identify individual users, based on recorded use of the system. If traces are to be shared and disseminated for research purposes, care should be taken to ensure that traces are anonymized.
!!!!Hybrid Approaches
Incorporating multiple approaches is an effective way to characterize and evaluate an entire system (<<cite Brown1997>>, <<cite Traeger2008>>). However, it can be difficult to relate results gathered from disjoint experiments performed at different levels of granularity; as such, structured hybrid methodologies may be more useful.
<<cite Seltzer1999>>'s //hybrid methodology// combines trace- and vector-based methodologies for application-specific benchmarking, using a simulator to identify system primitive operations and results used in the trace (e. g. cache hits and misses), thereby converting an application trace into an application vector.
<<cite Liedtke1998>> proposed a similar idea for constructing a representative workload from a trace: //stochastic benchmarks// randomly sample recorded user activity at a suitable level of granularity. A drawback to the stochastic approach is the incomplete knowledge of initial and global system state, as initial system state when replaying a trace cannot be assumed to be identical to the state when the trace was recorded. As a result, replaying full or stochastically-generated user traces restricts the types of possible experiments one could perform.
!!!!Approaches and Challenges
In reading the aforementioned workshop and position papers (<<cite Bershad1992>>, <<cite Liedtke1998>>, <<cite Mogul1999>>, <<cite Seltzer1999>>), I came to understand which evaluation approaches were common, as well as how evaluation was discussed and prioritized within the operating systems research community. In addition, I was made aware of the challenges faced by practitioners in this domain.
The use of previously published benchmarking protocols and benchmark workloads is common in systems research, and it is often questionable as to the extent to which these transfer from one experimental context to another. It is similarly difficult to assess when a heavily-used benchmarking protocol or workload becomes obsolete. Portability is another major concern, especially when comparing performance between architectures or operating systems. An example is <<cite Chen1996>>'s comparison of three commodity operating systems, discussed below. A unique problem affecting reproducibility in systems research is the use of proprietary or commercial benchmarking tools, which tend to be less accessible or freely available (<<cite Traeger2008>>). A related issue is the inability to publish performance results of logged behaviour of commodity software or systems, as stipulated by some end-user license agreements (<<cite Barham2003>>, <<cite Seltzer1999>>), which makes reporting comparisons between research system and commodity systems difficult.
In considering these challenges, I gained another lens with which to examine the surveyed research papers.
!!!Performance Evaluation Survey
In the preceding section I reviewed the major approaches to performance evaluation prevalent in operating systems research, summarizing their advantages and disadvantages with resect to experimental design and the reporting of results, as well as issues of realism, portability, reproducibility. With these issues in mind, I have examined four research papers (<<cite Brown1997>>, <<cite Chen1996>>, <<cite Clark2004>>, <<cite Traeger2008>>) which involve these approaches to performance evaluation.
!!!!From a Set of Tools to a Methodology
Those intent on studying and comparing the performance of low-level system primitives must consider the granularity, portability, and statistical rigour of an evaluation protocol and the measurement tools involved. Microbenchmarking tools and protocols, as described above, affords such low-level analysis differs, differing from macrobenchmarking or kernel profiling approaches; the former doesn't isolate the many variables of interest required to construct a decomposable hierarchy, while the latter approach is often infeasible due to unavailable or proprietary kernel source code. <<cite Brown1997>>'s //hbench:OS// is a benchmarking methodology and test suite that revises <<cite McVoy1996>>'s //lmbench// suite, allowing researchers to study the hardware-architectural basis of operating system performance. In this section, I concentrate on <<cite Brown1997>>'s revision, as well as their results and observations. However, I first provide a short summary of the original //lmbench// suite.
!!!!!lmbench
<<cite McVoy1996>>'s //lmbench// is a microbenchmark suite for evaluating the basic low-level system primitives, examining the transfer of data between the processor, cache, network, and disk. The suite is intended to be widely available and highly portable across architectures and operating systems, including both uniprocessor and multiprocessor systems; although in 1996 it did not incorporate any multiprocessor-specific tests. Benchmark suites prior to ~McVoy and Staelin's were described as being either too focused on a single low-level feature, not portable, not widely available under public license, had poor timing granularity, or either provided too many or too little tests, complicating subsequent analysis. The suite's tests, all run in main memory, recorded measurements of latency and bandwidth:
*The //bandwidth// benchmarks measure the rate at which a particular facility can move data, including memory read, write, and copy, as well as IPC and cached I/O.
*The //latency// benchmarks, measured at a clock tick granularity, include several measurements of memory, system calls, signal handling/CPU interrupt, process creation, IPC, file system operations, and disk operations.
The //lmbench// benchmark suite addresses cache and memory size issues by increasing the size of data used by factors of two, thereby measuring cache and memory performance separately. To account for variability, benchmarks are run multiple times and only the minimum result is recorded. To account for uncertainty with regards to whether the data is in the cache, the benchmark is run several times and only the last result is recorded.
!!!!!hbench:OS
The contribution of <<cite Brown1997>>'s //hbench:OS//, beyond the //hbench:OS// suite, is a portable benchmarking methodology, whereas they argue that //lmbench// is simply a set of tools without operational guidance. //hbench:OS// is an improvement upon //lmbench// in that the tests are more rigorous, self-consistent, reproducible, and conductive to statistical analysis, thereby being easier to analyse. It should be noted that in later versions of //lmbench// (<<cite Staelin2005>>), many of Brown and Seltzer's revisions were integrated into the suite, improving portability and extensibility.
Modifications to //lmbench// included updating the timer resolution with hardware cycle counters, removing overheads introduced by the timing mechanism. They also divorced data analysis from the data measurement components of the benchmark. Rather than taking a minimum measurement, //hbench:OS// takes a n% trimmed mean, discarded both the worst and overly optimistic values. This is useful when results are not normally distributed, such as a bimodal distribution. Cache priming is also done by running one iteration of the test before collecting data. The //hbench:OS// tests are also more parameterizable than //lmbench//'s tests, allowing for distinctions between dynamically and statically linked processes. Modifications to the memory read, write, and copy bandwidth tests to allow for measurement of the L1 and L2 caches separately. Context switching latency was not highly portable in the original //lmbench// suite, due to its inability to detect cache conflicts, a common microbenchmark problem also observed by <<cite Bershad1992>>. As a result, //lmbench// sometimes reported negative or zero-sized context switch latencies. //hbench:OS// measures only the true context switch time, approximated from cache and memory read bandwidths. They retain the original //lmbench// context switch test, noting that the new //hbench:OS// test has standard deviations that are lower (3% vs. 10%<). Finally, memory bandwidth tests were modified such that direct and indirect memory referencing could be more easily compared.
<<cite Brown1997>>'s //hbench:OS// methodology aims to characterize the whole system using a bottom-up approach, from subsystem bandwidth performance to the operating system and application level, where one must consider the latency of system calls, process creation, as well as file and network access. Thus the performance can be decomposed while varying features of the hardware, and features of one layer can be related to those at a layer above or below it: from hardware capabilities to low-level kernel primitives to high-level operating system services and finally to application performance. When described in this way, it is possible to see how //hbench:OS// methodology combines microbenchmarking and microbenchmarking, such that results of one can be directly attributed to results of the other. Naturally, a total reconstruction is not always possible, and middle levels of the performance hierarchy may be inaccessible to measurement or adjustment; these can be bypassed, leaving only operating system-dependent application performance at the top, and hardware capabilities at the bottom of the hierarchy. In these cases, if the hardware can be varied in a controlled manner, thus still providing some useful information relating the top and bottom of the hierarchy.
The authors report a case study of //hbench:OS// on eight machine configurations running the ~NetBSD 1. 1 operating system over the Intel x86 architecture. They measured bulk data transfer bandwidth, a bottom up analysis from the hardware level to the kernel and application levels, as well as process creation latency, a top-down analysis involving both static and dynamically linked processes. The bulk data transfer scenario is representative of bandwidth-sensitive applications such as web servers and multimedia applications, incorporating reading files, sending and receiving data via TCP, mapping files into an address space, where memory accesses were decomposed into memory reads, writes, and copies. With these decomposed results, the authors were able to predict performance at the operating system and application level. Overall, their case study found that despite CPU optimizations, hardware-level features of the memory system dominated operating system and application-level performance. Where measurements differed from those predicted from top-down analysis, it became possible to account for these differences, by isolating and examining optimizations or flaws in the hardware, in one case discovering that some kernel-level primitives did no depends on memory hardware performance.
What is important to retain from this article is that prior system evaluation benchmark tools, such as the original //lmbench//, forced a choice between levels of analysis, without providing a methodology or the guidance required to resolve results collected from hardware profiling, system primitive microbenchmarking, and application-level macrobenchmarking. //hbench:OS// introduced an evaluation alternative that was methodology-driven, rather than tool-driven, facilitating reporting and analysis of results while retaining portability and flexibility.
!!!!The Challenge of Comparative Evaluation
<<cite Chen1996>> compared the performance of three commodity operating systems running over a Pentium architecture. This article highlights the challenges of evaluation granularity, the identification of comparable cross-platform metrics, and the portability of evaluation methodologies and benchmark workloads. The authors measured system performance at the low level of system primitive operations using microbenchmark protocols as well as at the level of individual applications with representative workloads. Windows (for Workgroups), Windows NT, and ~NetBSD Unix are three commodity operating systems that are comparable because they are widely available and support the same typical patterns of use. However, the systems differ substantially at the kernel implementation level, with Windows lacking protected address spaces, preemptive multitasking, as well as high-level system abstractions such as pipes and background jobs. Due to these differences in system configuration, a fair low-level evaluation was difficult to design, as many preexisting microbenchmark suites cannot produce results that are comparable given these differences. In addition, the authors express their frustration with previously published benchmarks protocols and workloads, predating other who have made similar admissions (<<cite Mogul1999>>, <<cite Seltzer1999>>, <<cite Traeger2008>>). Chen et al. argue that these benchmarks, originally intended for descriptive commercial purposes, are misleading or incomplete, and cannot often be used for accurately answering comparative or predictive research questions.
Their methodology involved first gathering, sequencing, and running a set of microbenchmarks. Then, descriptive performance results were collected for a small number of cross-platform applications, which were selected based on the expected load they placed on the system primitives assessed using the microbenchmarks. The application performance was subsequently interpreted, again with regards to the microbenchmark results.
Their microbenchmark testing suite was largely based on the //lmbench// suite (<<cite McVoy1996>>), however this predated the //hbench:OS// revision contributed by <<cite Brown1997>>; Chen et al. 's study was likely a motivating factor behind the development of //hbench:OS//. Chen et al. contribute a few new measurements to the //lmbench// suite (noted in the following list); these were particularly useful for informing the design of the application-level benchmarks. Their microbenchmark suite contained the following tests:
*A null test: the baseline overhead latency to access the hardware counter itself (not in the original //lmbench// suite).
*The latency of a system call, an indication of the cost of accessing functionality implemented in the system.
*Running a trivial program, averaged over 50 invocations.
*A test of memory access time, referencing a large array with a stride of 128 bytes.
*A mix of file system operations, reflecting actual usage: accessing files that hit in the disk cache, accessing small files and go to disk, and file creation.
*A graphics subsystem benchmark (not in the original //lmbench// suite), which allowed them to inform and understand the //ghostscript// application workload (see below).
*A network throughput test, informing the web server application workload (see below).
The dependent performance measures for these microbenchmark tests were collected with calls to event and cycle hardware counters, accessible via device driver-level kernel extensions. The low-level events that could be observed and measured included data reads and writes, instruction reference misses in the cache and TLB, data reference misses in the cache and TLB, segment register loads, instructions executed, cycle counts, and hardware interrupts.
The differences between the three operating systems limited what could be compared at the microbenchmark level of system behaviour. In particular, Chen et al. were not able to discern and compare events logged at different protection levels, so they were unable to attribute performance results to user- or kernel-level events. The systems also differed in terms of how idle time was measured; as a result some time-based metrics in the //lmbench// microbenchmark suite could not be used. Instruction formats were hard to compare across the three operating systems; while instruction counts can be useful for comparing total work on RISC processors, the Pentium architecture on which they performed their experiments allowed for multi-cycle instructions, which were used differently in the implementation of Windows, Windows NT, and ~NetBSD. Since instruction counts were not always interpretable, they opted to instead to use cycle counts for comparing the total latency of computations. A drawback to this compromise was that they couldn't easily assign cycles to specific events. Finally, the different cache and TLB policies of the three operating systems made it difficult to absolutely compare the read, write, and miss metrics, however relative performance could still be discerned from the results.
At the level of application workload performance, applications were chosen based upon their compatibility with all three operating systems, and the belief that their performance could be explained by the microbenchmark results. Three applications were selected:
*''wish'': a //tcl-tk// command interpreter; a CPU intensive application with heavy use of the windowing system, requiring many context switches between application and the graphics server.
*''ghostscript'': a postscript viewer application, which also placed heavy demand on the windowing system; //ghostscript// is a a dynamically linked application, and was informed by graphics subsystem microbenchmark.
*A web server application that placed heavy demand on the network and file systems.
Application behaviour was somewhat predictable, in that each had a working set under 32MB and caused no significant paging during execution. During the application workload tests, system background activity was limited, the network subsystem was disabled except in the tests that required network access, and used single-user mode was enabled. Controlling these factors allowed for more precise measurements, at the cost of realism. Arguably the authors could have conducted a larger factorial experiment, thereby isolating the cost of these factors and measuring realistic performance; the current results could have served as a baseline control condition. On the other hand, this choice would have complicated the design and reporting of the experiment. Finding applications that were compatible with all three platforms also proved to be a struggle, as the authors were limited to open source applications. This constraint meant that the results may not generalize to a wide range of applications, and are likely biased toward the ~NetBSD system, since the applications used in the evaluation were originally developed for UNIX systems.
Despite the challenges Chen et al. faced in designing these experiments, as well as in recording and interpreting results, their aggregate results did convey apparent relative performance differences between the three operating systems. They found that Windows for Workgroups, which does not have protected address spaces, performed worse than expected. As a result, the authors questioned the unified address space model which was showing promise at the time, such as in the exokernel project (<<cite Engler1995>>). In addition, they observed Windows' use of "hooks", a mechanism that intercepts system calls, intended for application flexibility and backwards compatibility. These hooks further contributed to low-level performance differences between the operating systems, making comparisons more difficult.
This article demonstrates several of the challenges and tradeoffs prevalent in operating system evaluation. It raises questions about the portability of widely available benchmark protocols and workloads such as //lmbench//, and whether there exists a limitation to the type of comparative evaluation experiments one could perform. Due to the differences in operating system primitives and the availability of cross-platform applications, it's not surprising that making realistic and accurate comparisons proved to be a difficult endeavour.
!!!!The Challenge of Reproducibility
<<cite Mogul1999>> and <<cite Clark2004>> both advocate that more research effort on realistic and reproducible performance evaluation will result in the increased deployment and adoption of novel systems, believing that "the ability to accurately predict performance [will] translate directly into higher profits" (<<cite Mogul1999>>). There are several ways to make a system performance experiment more reproducible. The system's source code should be well documented and made available to other researchers; benchmark protocols and workloads should also be open, along with their testing scripts and parameter settings.
An example of repeated systems research was a project performed by <<cite Clark2004>>, who in 2004 successfully reproduced the performance results of <<cite Barham2003>>'s SOSP 2003 paper //Xen and the art of virtualization//, using nearly identical hardware. In addition, Clark et al. asked if the performance claims made in the //Xen// paper regarding scalability could be replicated using cheaper commodity hardware. //Xen// was designed to provide isolation and scalable performance for up to a target of 100 guest operating systems. Furthermore, these guest operating systems would be largely unmodified commodity operating systems running potentially thousands of industry standard applications. Clark et al. also inquired about the portability of //Xen//, whether it could be run on cheaper commodity hardware, less than $2,500, comparing how multiple guest operating systems running as guests over //Xen// on a commodity PC compares to Linux virtualized an IBM //zseries// mainframe (valued near $200,000 in 2004). This question meant extending the original //Xen// research protocol, adding an additional comparison across different hardware installations.
Reproducing the prior results from <<cite Barham2003>> required assembling and running all the benchmarks used in the original //Xen// paper, which included the //lmbench// microbenchmark suite (<<cite McVoy1996>>), writing the necessary scripts and setting parameters for these benchmarks. While much of the information required to reproduce the original results could be gleaned from the published //Xen// article, some parameters and finer points of the evaluation protocol were not fully specified; Clark et al. were fortunate in that the //Xen// authors were willing to divulge this information in private correspondence. However, not all of the benchmarks could be acquired; in the case of the proprietary [[SPECweb99|http://www. spec. org/web99]] web benchmark, an analogous benchmark was built to simulate it using a trace measurement tool. In reproducing the original scalability evaluation (<<cite Barham2003>>, ൩, Clark et al. discovered that Barham et al. only allocated 15MB for each guest, which Clark et al. described as being not realistic nor sufficient for an industry standard web server. A 128MB memory size per guest would be more typical, however this would require over 12GB of memory. Clark et al. stated that an upper bound of 16 guests is more realistic for a system with 4GB of memory. As a result, Clark et al. allocated 98MB per guest operating systems, whereupon they observed that //Xen// was able to successfully scale to 16 guest operating systems.
Clark et al. also observed that the original //Xen// evaluation compared against Linux with SMP (Symmetric Multiprocessor System) support disabled in some but not all experimental conditions. This choice in parameter settings may have accounted for some of Linux's relative performance in the results presented in the original //Xen// paper (<<cite Barham2003>>). In all conditions, Clark et al. compared Xen against Linux both with and without SMP support.
With regards to Clark et al. 's question of whether //Xen// could be used on an older commodity PC, they found that //Xen// could be successfully installed on such a system, albeit with a smaller number of guest operating systems; the performance of //Xen// on a commodity PC is similar to that of the Linux virtualization on an IBM //zseries// mainframe, which is remarkable considering the difference in cost is nearly two orders of magnitude.
Clark et al. 's article is a successful instance of repeated research: not only were previous performance results repeated, but they also pointed out inconsistencies and details absent in the original //Xen// article (<<cite Barham2003>>), in the scalability evaluation in particular. In addition, they extended the original research, posing and answering a new question about the portability and cost of a //Xen// installation. They conclude by arguing that researchers should strive to make their research more reproducible, as repeated research provides confidence to novel systems, encouraging technology transfer and industry adoption.
!!!!Guidelines, Pitfalls, and Claims Debunked
In the previous sections I have summarized the challenges associated with evaluating the performance of operating systems and subsystems, serving to illustrate the need for more guidance in this regard. <<cite Traeger2008>> presented such guidance, along with a survey of file system evaluation as reported nine years' worth of research papers from high-impact operating systems venues; in total, 106 SOSP, OSDI, and USENIX papers published between 1999 and 2007 were included in the survey. Specifically, they examined and compared performance evaluation methods and methodologies and the reasoning behind the use of benchmarks. This article contributes more than a descriptive survey; it is also prescriptive in that indicates shortcomings and strengths, and it provides guidance for future research regarding how to create and use benchmarks effectively, as well as how to present results.
Throughout the article, Traeger et al. also report on their own experiments, either reproducing the results or debunking the claims of the surveyed papers. In some cases, these experiments serve the purpose of illustrating methodological assumptions or insufficient reporting of experimental protocol in previous research. Their own experiments followed a benchmarking methodology in which each test was run at least 10 times; 95% confidence intervals were computed for mean elapsed, system, and user times using the student-t distribution. They disabled unrelated system services and rebooted the machine between successive sequences of benchmark runs, ensuring consistent cache states. They automated benchmark runs using the Autopilot benchmarking suite, a prior project of one of the authors. Throughout their survey, they demonstrate that different benchmarking decisions can hide overheads and latencies, particularly with regards to compile time macrobenchmarks.
While focused on file system evaluation, many of the observations and guidance presented in Traeger et al. 's article may be generally applicable to other areas of systems research; the authors position this paper in a broader context of related systems work, some of which discussed previously in this report (<<cite Mogul1999>> ,<<cite Seltzer1999>>). <<cite Mogul1999>> also surveyed system performance evaluation as reported in SOSP and OSDI papers prior to 1999. He observed a trend in which papers in a particular topic area, such as file systems, do not often share evaluation protocol and benchmark choices, nor is there an agreed-upon approach for reporting results. He also surveyed a similar-sized body of research from recent computer architecture publishing venues, observing that performance evaluation design, analysis, and reporting tended to be more consistent within that community. A more exhaustive survey of this nature can be found in an earlier technical report from Seltzer's research group (<<cite Small1997>>), which span topic areas and focuses on the lack of statistical rigour in the analysis and presentation of system performance results, motivating the work on application-specific benchmarking discussed earlier (<<cite Seltzer1999>>). In addition to the problems inherent to all systems-related evaluation, Traeger et al. mention that evaluating file and storage systems often requires extra care, in that these systems may interact in complex ways with other subsystems, and may differ from other subsystems in terms of their underlying media, their storage environment, and their expected workloads. I will not reproduce the descriptive elements of Traeger et al. 's survey; instead, my summary will focus on the empirical observations and prescriptive guidance contributed by the authors.
Above all, when evaluating a file or storage system, Traeger et al. insist upon //reporting what was done in as much detail as possible//, and //explaining why it was done that way//. Throughout their survey, they observed that many research papers seldom include both of these components, the former necessary for reproducibility and the latter necessary for understanding the intended contribution of the system or systems under study.
//What was done?//: detailed reporting of the experimental context and the state of the system should be reported: is the cache warm or cold? For disk storage benchmarks, where are the partitions located? Is the file system empty, or has the system been aged or subject to real-world use, and if so, for how long? Are other nonessential services running during the evaluation; what interactions occur? Are workloads multithreaded?
With regards to performance data collection and analysis, Traeger et al. compiled operational guidance for executing an evaluation protocol, reporting said protocol, and reporting results. Care should be taken such that each test run is identical, standard deviations and confidence intervals should be computed in the same way. Automated scripts may perform these tasks, thereby limiting human error. When reporting a protocol, one should include the number of benchmark runs, the benchmark runtimes, the number of benchmarks, and a description of the system state, including the state of the cache.
When reporting experimental results, confidence intervals are recommended over standard deviation, as the former produces a better sense of the true mean, generally decreasing as more test runs are performed; the standard deviation, on the other hand, captures the variation between successive runs, and may not decrease over time. When analyzing the results and computing statistical measures, a normal distribution can only be assumed for more than 30 runs; less than 30 is considered to be a small sample size, where a student-t distribution is more appropriate. Anomalous results, large confidence intervals or non-normal distributions should not be discarded; a software bug or erroneous benchmarking script may be the cause.
//Why was it done that way?//: the purpose of an evaluation should also be clear. One's intent may be to compare against other similar systems, to examine the performance when subjected to an expected workload, or to identify the causes of performance overheads or improvements. Of these, the first is most meaningful to readers, so it is often worthwhile to include a comparison to an alternative system in an evaluation whenever possible. The latter often requires testing several configurations in turn. It is also encouraged to evaluate both high-level and low-level performance, usually satisfied by either macrobenchmarks or a trace-based evaluation (high level) and a set of microbenchmarks (low level). It is often important to question the realism, granularity, accuracy, and scalability of the both macrobenchmarks and traces. Results provided by microbenchmarks are more meaningful when used to explain low-level performance differences, highlighting worse-case behaviour, isolating specific effects or interactions.
Next, Traeger et al. discussed the types of evaluations as reported in the papers in their survey. Here they discuss trends and common approaches for using and reporting on macrobenchmarks, trace-based evaluations, and microbenchmarks, defined above.
!!!!!Macrobenchmarks
Macrobenchmarks aim to simulate a real-world application workload, however many papers fail to describe the reasoning for opting to use a macrobenchmark. An example is the //Postmark// macrobenchmark, which uses a synthetic workload, but doesn't perform any actual application processing itself. The workload size is not configurable, so it doesn't scale to modern systems (as of 2008). Another inefficiency is //Postmark//'s file selection algorithm, which is O(N) on the number of files. Traeger et al. suggest that a configurable and accurately measurable run time would be more scalable than a configurable workload size, thereby affording better longitudinal comparisons as hardware improves.
Compile benchmarks are another flavour of macrobenchmark, however these vary considerably across architectures and compiler chains. The authors empirically debunk the assumption that file systems see similar workloads with compile benchmarks, independent of the software being compiled; in reality these workloads vary from run to run, even on the same machine.
Traeger et al. also point out that many popular research and commercial macrobenchmarks do not provide configurable operation mixes, and many suffer from from being outdated, being not reflective of modern application behaviour and their use of the cache. Commercial macrobenchmarks from the TPC, SPC, and SPEC organizations are not widely or freely available. The authors also observe that these benchmarks are often not run according to their specifications.
!!!!!Trace-based Evaluation
Replayable traces are also used in file system evaluation. While the recorded trace is a real-world workload, questions of generalizability must still be asked. Some papers recorded traces of macrobenchmarks, which is as questionable as the macrobenchmarks themselves, potentially negating the realism of the trace methodology; although, as some macrobenchmarks are proprietary or expensive, traces offer a compromise by providing a benchmark-generated workload. Traeger et al. observed, as <<cite Liedtke1998>> had a decade earlier, that there is no clearly-defined or agreed upon way to record and replay traces, nor are trace recording tools made available or reported in sufficient detail. Traces can be recorded at several levels, from the level of system calls to that of network or driver protocols, requiring a tradeoff decision between trace granularity and portability.
Trace replaying is likely to occur at the level it was recorded; this may involve aging a file system before replaying, and this process is seldom explained. Replay speed should also be justified: often, a trace is run either at the speed it was recorded, or it might be run as fast as possible.
Finally, traces should be made available for promoting reproducible research, however precautions must be taken to ensure the anonymity of trace data.
!!!!!Microbenchmarks
Microbenchmarks test a small number of low-level operations, highlighting performance overheads or benefits implied by macrobenchmark or trace results, or to isolate a specific aspect of the system. Popular microbenchmarks share many of the same considerations described above for macrobenchmarks: scalability, cache effects, generalizability, portability, configurability, and accuracy. Some microbenchmarks are trivially simple to reproduce, even if not publicly or freely available, slight variations in microbenchmark implementations can lead to significant differences in results; Traeger et al. empirically show how five subtly different implementations of the //Sprite LFS small-file benchmark// produce significantly different results.
Ad hoc single-use microbenchmarks are the most difficult to reproduce and are most prone to bugs, however these may still be useful in conjunction with other microbenchmarks, such as for explaining some unexpected or anomalous result. Ad hoc single-use microbenchmarks may also be useful in the initial phases of benchmarking, to explore the behaviour of the system, to guide the selection of more widely accepted and available benchmarks.
Another form of microbenchmarking is the use of standard system utilities as a means of creating a representative workload, such as wc, grep, cp, diff, tar, and gzip. These utilities are widely available and understood; however, versions of these utilities are subject to change, and they may not scale, given different input files.
!!!!!Unmet Needs
Traeger et al. 's findings highlight a need for empirical methods that allow for absolute and relative comparisons of multiple workloads. Methods are also required for normalizing performance results for the hardware and operating system on which they were collected, thereby facilitating cross-system comparisons.
!!!Discussion
Given the common interrelated challenges associated with the various approaches to system evaluation described above.
''Realism/Representativeness'': in retrospect, <<cite Seltzer1999>> have admitted that even a system benchmark such as //hbench:OS// (<<cite Brown1997>>) may not provide realistic indicators of system performance specific to particular applications. While <<cite Traeger2008>>'s article pertained to file system research, the challenges discussed were shared by the other papers, which dealt with other topic areas. As such, it is possible that some of the methodological guidance presented in that article is transferable. According to Traeger et al. , triangulating on a representative indication of system performance may be possible by combining a hierarchical system benchmark such as //hbench:OS// (<<cite Brown1997>>) with a trace-based approach or a //stochastic benchmarking// approach (<<cite Liedtke1998>>), recording and replaying a trace of operations. Another option is combining benchmarking with application workload testing, as in the case of <<cite Chen1996>>, however in that study, realism was hampered due to the choice of UNIX-centric applications. If one is interested in the performance of a particular set of applications, the //hybrid-based approach// (<<cite Seltzer1999>>) described above, where an application trace is converted to an application vector, subsequently combined with a system primitive vector, resulting in an indication of overall performance.
Despite a mixture of evaluation methods, unrealistic experimental protocols, workloads, and parameters may not be initially apparent when reading a research paper, and these concerns may only become apparent upon reproducing the work. Such was the case with <<cite Clark2004>>'s reproduction of the earlier Xen paper (<<cite Barham2003>>), who found that Barham et al. 's scalability experiment was not representative of typical web server configurations.
''Granularity of Measurement'': the //lmbench// (<<cite McVoy1996>>), used by <<cite Chen1996>> in their evaluation of three commodity operating systems, was developed in response to the imprecise granularity of prior benchmarks, whose measurements did not capture system primitive performance and yet neither were they representative of application-level performance. //hbench:OS// (<<cite Brown1997>>) took this a step further, providing means to measure system performance from the hardware level to the application level, with fine control over the granularity of timing measurement. Combining evaluation approaches is another means to ensure that performance measurements are observed at several levels of granularity, using microbenchmarks and macrobenchmarks, as advocated by <<cite Traeger2008>> and as reported by <<cite Clark2004>>, who reproduced earlier work by <<cite Barham2003>>.
''Comparability/Portability'': While <<cite Traeger2008>> suggested many ways to improve the comparability of results across systems and over time, facilitating relative comparisons, they saw the task of performing accurate comparisons as being an open question deserving of future work. The works of <<cite Chen1996>> and <<cite Clark2004>> were immediately concerned with the comparability of results and the portability of experimental protocols and parameters; the former attempted this within the same experiment across three commodity operating systems, while the latter aimed to compare against prior work (<<cite Barham2003>>) using different underlying hardware.
''Reproducibility/Transferability'': Many of the papers surveyed speak to the need for reproducible results and transferable evaluation methodologies. This was an overarching motivation for the development of //hbench:OS// (<<cite Brown1997>>), described as being more than just a set of tools but also a methodology. <<cite Traeger2008>> stressed the need to specify one's methodology, documenting //how an evaluation was performed//. They lead by example, documenting their own experiments reported throughout their article in great detail. Interestingly, <<cite Chen1996>> do not speak about reproducibility in their article; relative to the other papers surveyed, I found the description of their experimental protocol to be the most imprecise and lacking in rationale; it would likely be difficult to transfer their hardware-specific methodology to three other operating systems. Finally, the importance of repeated research is no less stated than in <<cite Clark2004>> article, whose principal contribution is a successful reproduction of prior research findings, which also served to reaffirm the strength of the original work (<<cite Barham2003>>).
!!!Conclusion and Future Work
A line of potential research projects could involve compiling evaluation guidance particular to specific topic areas within systems literature, akin to <<cite Traeger2008>> survey of recent file systems research. A new survey of recent papers with respect to statistical rigour in performance evaluation might also be of value, especially if compared to similar work performed in the late 1990s (<<cite Small1997>>). Development of new experimental protocols is also promising direction of future work, an aim of which should be to address the unmet needs identified above.
In this paper I presented a survey of four research papers pertaining to operating system performance evaluation. I summarized the approaches to system evaluation, which included micro- and macro-benchmarking, trace-based evaluation, application-based evaluation, as well as combined or hybrid approaches.
The surveyed position, workshop, and research papers illustrated the prominent challenges and tradeoffs associated with evaluation: realism, the granularity of measurement, the portability of experimental protocol, as well as the comparability and reproducibility of methods and results. The existence and impact of the papers surveyed have demonstrated that systems research, being representative of the nascent field of computer science, is moving toward a rigorous and standardized science of measurement and evaluation.
!!Proposal
I'd like to propose a survey paper focusing on performance evaluation in operating systems research. As background context for this interest, my research in HCI relates to the development of evaluation methods and methodologies, with a particular focus on user-centred evaluation of information visualization tools and techniques. However, I'm more broadly interested in how computing techniques and systems are evaluated in other areas. I want to better understand what the operating systems research community considers to be a rigourous analysis of system performance, as well as how the results of these analyses ought to be reported in publications. I am also curious as to how the science of empirical performance evaluation has developed over time, in response to developments in operating systems for personal and distributed computing. I will be particularly attentive to issues of methodological reproducibility and transferability, as well as external validity, identifying representative comparisons, workloads, and benchmarks with average- and worst-case real-world scenarios in mind.
Additional inspiration for this proposed survey came upon reading Cao et al. 's LRU-SP paper (1996) and Engler et al. 's Exokernel paper (1995). Both contained a quantitative evaluation of an implemented system's performance, comparing against alternative existing systems. I have since been browsing the operating systems literature for similar and different evaluation methodologies. I acknowledged that this scope covers a large literature space, so my process thus far has been to narrow down my search: collecting highly-cited literature from the SIGOPS, SIGMETRICS, and USENIX communities, particularly those papers whose primary research contributions is a performance evaluation. While I'd be most interested in taking a historical approach [R 1-7], an alternative could be to review performance evaluation for a particular area of systems research, such as distributed file systems [R 8-10]. I would welcome your feedback on these alternatives and I am happy to iterate on the selection of references.
Finally, with regards to reporting and reviewing evaluation papers, do guidelines or resources exist? For instance, I've looked to calls for papers on systems research publication venues (OSDI, SOSP), where it is mentioned that good papers "will demonstrate the practicality and benefits of the solution", among other criteria; how should this be demonstrated effectively, and what should reviewers look for? Please let me know if you are aware of any such resources.
To sum up, I hope that this survey will give me an opportunity to explore the development of a rigourous empirical science of evaluation in an area that is quite different from my own research. //As motivation, I know from reading material relating to the philosophy of science that great discoveries are often made by analogy, with methodological inspirations emanating from radically different fields. //
!!!Feedback:
>''MF'': //Interesting topic, but a bit difficult to suggest papers for. //
>//Take a look at the following://
>>[1] D. Roselli, J. R. Lorch, and T. E. Anderson, ﭰarison of file system workloads,퐲oc. USENIX Annual Technical Conf. , 2000.
>>
>>[2] B. Clark, T. Deshane, E. Dow, S. Evanchik, M. Finlayson, and J. Herne, d the art of repeated research,퐲oc. USENIX Annual Technical Conf. , 2004, pp. 135>>
>>[3] G. Baker, J. H. Hartmart, J. K. Ousterhout, and K. W. Shirriff, 㵲ements of a distributed file system,퐲oc. ACM Symp. Operating Systems Principles (SOSP), 1991, pp. 198>>
>>[4] W. Vogels, 堳ystem usage in Windows NT 4. 0,퐲oc. ACM Symp. Operating Systems Principles (SOSP), 1999, pp. 93>>
>>[5] M. Seltzer, D. Krinsky, and K. Smith, ࣡se for application-specific benchmarking,퐲oc. WS. Hot Topics in Operating Systems, 1999, pp. 102>>
>//Your other papers are quite a broad mix. Note that #8 isn't really so much a measurement paper, despite its name. //
>
>//SIGMETRICS is THE measurement venue for systems stuff. . . //
>
>//SOSP is THE conference for systems in general, with OSDI #2 (see also specialized conferences like NSDI). Generally there is a conference version of papers that appear in TOCS, which will be shorter :). //
!!!References:
!!!!Historical background:
[1] Calingaert, P. (1967). System performance evaluation: survey and appraisal. Communications of the ACM, 10, 12-18.
[2] Cantrell, H. N. and Ellison, A. L. (1968). Multiprogramming system performance measurement and analysis. In Proc. Spring Joint Computer Conf. (AFIPS), 213.
[3] Heidelberger, P. (1984). Computer performance evaluation methodology. IEEE Trans. Computers, C-33, 1195-1220.
!!!!The past 20 years:
[4] Gupta, A. , Tucker, A. and Urushibara, S. (1991). The impact of operating system scheduling policies and synchronization methods of performance of parallel applications. ACM SIGMETRICS Performance Evaluation Review, 19, 120-132.
[5] Chen, J. B. and Bershad, B. N. (1993). The impact of operating system structure on memory system performance. In Proc. ACM Symp. Operating Systems Principles (SOSP), 120-133.
[6] Chen, J. B. , Endo, Y. , Chan, K. , Mazi전. , Dias, A. , Seltzer, M. and Smith, M. D. (1996). The measured performance of personal computer operating systems. ACM Trans. Computer Systems, 14, 3-40.
[7] Chou, A. , Yang, J. , Chelf, B. , Hallem, S. and Engler, D. R. (2001). An empirical study of operating systems errors. In Proc. ACM Symp. Operating Systems Principles (SOSP), 73-88.
!!!!Performance evaluation in distributed file systems:
[8] Howard, J. H. , Kazar, M. L. , Menees, S. G. , Nichols, D. A. , Satyanarayanan, M. , Sidebotham, R. N. and West, M. J. (1988). Scale and performance in a distributed file system. ACM Trans. Computer Systems (TOCS) 6, 51-81.
[9] Spasojevic, M. and Satyanarayanan, M. (1996). An empirical study of a wide-area distributed file system. ACM Trans. Computer Systems, 14, 200-222.
[10] Thereska, E. , Salmon, B. , Strunk, J. , Wachs, M. , Abd-El-Malek, M. , Lopez, J. and Ganger, G. R. (2006). Stardust: Tracking activity in a distributed storage system. ACM SIGMETRICS Performance Evaluation Review, 34, 3.
!!Notes
!!![<<cite Seltzer1999>>] ~Application-Specific Benchmarking
<<cite Seltzer1999>> argue that system-primitive-level microbenchmarks that do not consider application usage patterns and representative loads on these primitives are of little use to fellow researchers or potential adopters of those systems. On the other hand, application-specific benchmarks are often not reproducible or allow for meaningful comparisons across alternative systems or over periods of time.
The paper proposes three transferable methodologies for application-specific benchmarks, as it is impractical to develop a new benchmarking methodology for each evaluation of an application on a particular system at a specific point in time:
*''A vector-based methodology'': a system characterization vector accounts for the underlying systems primitives, which is then combined as a dot product of a application vector that represents the demand an application places on each primitive. The result is system performance. Advantage: a general model that leverages any well-defined system API used by an application. Disadvantage: difficult to capture garbage collection in the JVM, relative performance is often correctly measured, but not absolute performance, doesn't capture particular streams of application usage.
*''A trace-based methodology'': usage loads are modelled after particular streams of application use; advantage: run what-if scenarios, mimic anticipated loads, loads are derived from logs, which are often proprietary, whilst trace-based usage loads are not. Disadvantage: loads are specific to an application/system combination, doesn't give insight into underlying system primitives seen in vector analysis.
*''A hybrid methodology'': combines trace and vector approaches, uses a simulator to convert an application trace into an application vector, identifying operations / primitives used in the trace (e. g. cache hits and misses). Advantage: combines advantages of both approaches. Each have advantages and disadvantages, constraints and requirements.
!!!!Comments & Questions
*The examples used in the paper dominantly come from research on file system designs. Could this be applied for comparing application-controlled memory policies as in AFCS? What if system primitives / policies are novel? Would trace-based approaches make meaningful comparisons?
*Note: check forward refs, who uses these methodologies?
!!![<<cite Clark2004>> Xen and repeated research
<<cite Clark2004>> reproduce the results of Barham et al's //Xen and the art of virutalization// SOSP '03 paper [Xen03], using nearly identical hardware. In addition, they ask if the claims made in the Xen paper regarding its scalability for web hosting are possible. They also inquire if Xen could be run on cheaper hardware (less than $2. 5K), and compare how multiple OSes running over Xen on a commodity PC compares to Linux virtualized an IBM mainframe.
Reproducing the prior results from [Xen03] required assembling and running all the benchmarks, writing the necessary scripts and setting parameters for these benchmarks. In the case of the proprietary SPEC INT SPECweb99 web benchmark, an analogous benchmark was built to simulate it using Apache JMeter. Clark et al allocated 98MB per guest for up to 16 guests.
The authors observe that the [Xen03] evaluation compared against Linux with SMP (Symmetric Multiprocessor System) support disabled in some experimental conditions, which may have accounted for some of Linux's relative performance in the results presented in [Xen03].
Clark et al show that Xen can be used on an older PC, but only with a smaller number of guests; the relative performance to Linux is similar. Xen outperforms Linux virtualization on an IBM zseries mainframe ($200K).
Repeated research is good for reproducibility. Code should be open source, benchmarks should be open, along with their testing scripts and parameter settings.
!!!!Comments & Questions
>//"it is most common for researchers to report results from testing the software that they themselves have implemented"//
*Mention the importance of presenting standard deviations of benchmark runs to demonstrate reliability (not reported in [Xen03]) - they used 5 runs
*Notes for Xen:
**128 guests w/ 15MB each is very small, not sufficient for an industry standard webserver. A typical size is 128MB. For 100 guests, over 12GB is required. On a Xeon server, only 4GB is available ( 20 guests). Clark et al evaluate with up to 16 guests. They would not expect 100 guests running industry standard applications to be feasible.
**Linux kernel supports 15 partitions. [Xen03] patched the kernel to allow 128 raw disk partitions.
**They flip-flopped between running Linux with and without SMP support.
!!![<<cite McVoy1996>>] lmbench
McVoy and Staelin's ''lmbench'' is a microbenchmark suite for evaluating the basic low-level components (building blocks) of a system: transferring data between processor, cache, network, and disk. The suite's metrics are ''latency'' and ''bandwidth''. The suite is intended to be widely available and highly portable across architectures and operating systems, including both uniprocessor and multiprocessor systems, though it does not incorporate any multiprocessor features.
Other benchmark suites (// IOstone, BSD microbench, Ousterhout's OS benchmark, Netperf// and //ttcp// networking benchmarks) are either too focused on a single low-level feature (not all-encompassing), not portable, not widely available under public license, or either provided too many or too little tests. McCalpin's //stream// benchmark is an exception, and there are plans to integrate this into the lmbench.
The //lmbench// benchmark addresses cache and memory size issues by increasing the size of data used by factors of two, thereby measuring cache and memory performance separately. All benchmarks are run in main memory.
Timing resolution is accounted for at the clock tick granularity. To account for variability, benchmarks are run multiple times and only the minimum result is take. To account for uncertainty with regards to whether the data is in the cache, the benchmark is run several times and only the last result is recorded.
Bandwidth benchmarks: rate at which a particular facility can move data
*memory bandwidth: unrolled and libc bcopy (rather than cache bandwidth, data sized accordingly), memory read and write
*IPC bandwidth: pipe / TCP (costs associated with bcopy, checksum, network interface driver)
*cached I/O bandwidth: overhead of reusing data, libc bcopy, file read, memory read, file mmap
Latency benchmarks: often overlooked, rarely accurately measured, frequently misunderstood. latency improved by shortening paths, better prefetching. Different types of latency include memory chip cycle latency, pin-to-pin latency, load-in-a-vacuum latency, back-to-back-load latency. Average latency will be closer to back-to-back than in-a-vacuum in common cases.
*memory load latency: level 1 and level 2 cache latency -> total memory latency
*operating system entry: (system calls)
*signal handling costs: interrupts to the CPU
*process creation costs: simple process creation (fork and exit)), new process creation (fork, exec, exit), running a different program
*context switching save state of one process, restore state of another; adjusting number and size of processes
*IPC latency: control messages b/w processes, round trip latency (pipe latency), TCP / RPC latency, UDP latency, network latency, TCP connection latency
*file system latency: time to create, delete a file
*disk latency: sequential and random disk I/O, memory-to-memory transfers.
A major finding of running the benchmarks on contemporary systems was that a good memory subsystem was at least as important as processor speed.
!!!!Comments & Questions
*It was used in //Xen and the art of virtualization// for evaluating the performance of Xen's low-level primitives:
**processes: null call, null I/O, stat, open/close, scly TCP, sig inst, sig hndl, fork proc, exec proc, sh proc
**context switching: 2p, 8p, 16p x 0k, 16k, 64k configs
**file and VM latencies: 0k file create/delete, 10k file create/delete, mmap, prot fault, page fault
*Systems evaluated were high-end uniprocessor and multiprocessor systems of the mid 90s*FW to include multiprocessor benchmarks, static and dynamic processes using shared libraries, better measures of memory latency
*not a significant amount of discussion / research-y bits, more of just a summary. They made a case for portability, coverage of latency and bandwidth issues, availability (free licensed). Did not argue against higher-level trace or vector-based benchmarks, application-specific benchmarking. Issues of scalability not addressed.
!!![<<cite Brown1997>>] hbench:OS
//hbench:OS// is revision of //lmbench//, intended to study the hardware/architectural basis of OS performance as a decomposable hierarchy. This is opposed to macrobenchmarking or kernel profiling approaches, as the former has too many variables and with the latter one is often faced in unavailable kernel source code.
The contribution, beyond //hbench:OS//, is a benchmarking methodology, whereas //lmbench// is just a set of tools. //hbench:OS// is an improvement upon //lmbench// in that the tests are more rigorous, self-consistent, reproducible, and conductive to statistical analysis (easier to analyse).
Modifications to //lmbench// included updating the timer resolution (hardware cycle counters) and ensuring that destructive measurements can only be taken once, removing overheads introduced by the timing mechanism. They also divorced data analysis policies from the benchmark itself. Rather than taking a minimum measurement, //hbench:OS// takes a n% trimmed mean, discarded both the worst and overly optimistic values. This also helps when results are not normally distributed (e. g. bimodal). Cache priming is also done by running one iteration of the test before collecting data. The //hbench:OS// tests are also more parameterizable than //lmbench//'s tests, allowing for distinctions between dynamically and statically linked processes, or modifications to the memory read/write/copy bandwidth tests to allow for measurement of the L1 and L2 caches. Context switching latency was not highly portable in the original //lmbench// suite, due to its inability to detect cache conflicts, which sometimes resulted in negative or zero-sized latencies; //hbench:OS// measures only the true context switch time, approximated from cache and memory read bandwidths. They retain the original //lmbench// context switch test, noting that the new //hbench:OS// test has standard deviations that are lower ( 3% vs. >10%). Finally, memory bandwidth tests were modifies such that methods or accessing memory (read, touch, write) could be more easily compared, by converting direct and indirect references to memory.
The associated methodology aims to characterize the whole system, from hardware capabilities (e. g. memory bandwidth) to OS to application level (system calls, process creation, file/network access). Thus the performance can be decomposed while varying features of the hardware, and features of one layer can be related to those at a layer above or below it. Similarly, total system performance can be reconstructed. A total reconstruction is not always possible, and middle levels of the performance hierarchy (low-level kernel primitives, high-level OS services and primitives) may be bypassed, leaving only OS-dependent application performance at the top and hardware capabilities at the bottom. In such cases, if the hardware can be varied in a controlled manner, thus still providing some useful information relating the top and bottom of the hierarchy. In these cases, the results are still generally useful.
The authors report a case study of //hbench:OS// on 8 machines running the ~NetBSD 1. 1 OS. They measure bulk data transfer bandwidth (bottom up analysis from hardware, kernel, and application levels), process creation latency (top-down analysis, measuring both static and dynamically linked processes), and signal handler installation. The bulk data transfer scenario is representative of bandwidth-sensitive applications such as web servers and multimedia/network video applications, incorporating reading files, sending and receiving data via TCP, mapping files into an address space (memory accesses decomposed into hardware memory read, write, copy). Using these results, they can predict performance at the OS/application level. Where measurements differ from predicted, there may be optimizations or flaws in the hardware, indicative that the kernel-level primitive may have non-memory system dependent components.
Overall, their case study found that despite CPU optimizations, the memory system dominates performance.
!!!!Comments & Questions
*note: can't find any macrobenchmarking (end-to-end) / kernel profiling references
*referenced in Xen, by then lmbench was revised to incorporate the elements described in this paper
*systems analyzed in case study current as of 1997
*in non-decomposable scenarios such as in ೠw/ signal handler installation, the benchmark results may indicate that their is a problem, but the problem is still somewhat of a black box.
*<<cite Seltzer1999>> sez: //hbench:OS// is useful for comparing low-level primitives, but does //not provide an indication of which aspects of the system's behaviour are important for a particular application//.
!!![<<cite Liedtke1998>>] on irreproducible benchmarks (position)
This position paper begins by asking: why benchmarks? Benchmarks are used in research to understand a complex system: to characterize or predict the effects of a modification, rather than for commercial purposes. Benchmarks can help to identify bottlenecks, or (in)validate theory.
However, benchmarks break down because of non-transitivity (combining multiple optimizations proven separately by benchmarks may not have positive effects), instability (optimizations may have a positive effect on a benchmark but perhaps not on the applications that are similar to the benchmark), and non-linearity (different behaviour from concurrent and sequential benchmarks).
They discern between ''microbenchmarks'' and ''macrobenchmarks'', the latter having realism and analyzability. However, studying users using interactive systems is a problem for reproducibility. You can construct traces of behaviour, but it is not clear when to trigger recorded actions in response to system behaviour, and you can't account for how a user's behaviour changes in response to changing the system behaviour. User behaviour is inherently irreproducible and unpredictable. You can measure performance data as well as a user trace, the former is non-reproducible but the latter can be.
>//dropping the reproducibility increases realism and widens the applicability of benchmarks//
''Stochastic benchmarks'' attempt to attain realistic, representative data from randomly sampling recorded user activity. They are selected at random, tracing all (or some) levels of the system, at a granularity that is larger enough to label with semantic meaning without being intrusive enough to have the user notice them. A drawback to the stochastic approach is the incomplete knowledge of global state (nor can initial state snapshots be assumed to hold when replaying a trace).
Replaying full or stochastically-generated user traces restricts possible experiments. Replaying traces on modified hardware can still be informative. In OS research, the traces can be run against different policies (scheduling, page allocation) or can simulate realistic load. Traces could also be recorded or replayed in A/B comparison studies. However, changing the system may make recorded traces inapplicable.
!!!!Comments & Questions
*A position paper (no original research), not widely cited (why?) no follow on work from Liedtke et al?
*contrast with application-specific benchmarking (<<cite Seltzer1999>>): traces are only appropriate for application-system combination. Can you really stochastically trace the entire system and get something meaningful our of it?
**hybrid-based methodology of vector
*trace benchmarking may be more appropriate (or more actionable) than some of the proposed OS experiments mentioned here.
!!![<<cite Mogul1999>>] on brittle metrics (position)
The article argues that benchmarks are misleading and broken, leading researchers to study "what is easily measured". Benchmarks should be more realistic and applicable to real-world application domains. They should strive to measure absolute performance in a production environment (or at least relative performance or rank-order performance). Naturally, good benchmarks are hard to define and construct, particularly in OS research, where most of the intellectual effort goes into the implementation.
He goes on to review approaches to measuring performance on OS research: official research and vendor benchmarks, microbenchmarks, traces of real-user activity, and ad-hoc application combinations. Official benchmarks are easy to user and provide simple numeric results. Like <<cite Liedtke1998>>, tracing real user behaviour is good for realism, but changing the system often changes human behaviour; trace-based systems are becoming more important in mobile computing. Microbenchmarks can't measure the whole system. Ad hoc application selections might be fine if the system under measurement was designed with these particular applications in mind.
He also surveys OS research papers from SOSP and OSDI (1996), noting that many studying particular system components do not share often benchmark choices, particularly in file system research (whereas this is more so the case in computer architecture research).
>//the ability to accurately predict performance translates directly into higher profits.//
He advocates more research effort on getting novel systems deployed by means of realistic performance evaluation. There is a tension between realism and reproducibility. Traces are the future, complementing and contributing to better benchmark design: sharing research results will mean sharing traces. Anonymizing traces will be required. Benchmarks can give us realistic execution environment, traces give us a realistic request stream.
!!!!Comments & Questions
*a position paper, no original research, but a call - more widely cited than Liedtke
*contrast with application-specific benchmarking ( (<<cite Seltzer1999>>): trace-based + hybrid-based benchmarking methodology
!!![<<cite Bershad1992>>] on microbenchmarks (position)
A short position paper about the assumptions made by microbenchmarks: that they are representative of real programs, of something important on its own right, or have a measurable impact on system performance. Microbenchmarks often do not account for cache conflicts and cache effects: performance can vary from run to run, or between different versions of the same program when cache effects occur. They also ignore the side effects associated with write buffers, overestimating some primitives and using them in ways that wouldn't be reflected in practice. As a result, microbenchmarks are often misleading.
To make microbenchmarks more reproducible, they should explicitly flush the cache between trial runs, revealing worst-case, but reproducible, performance. Metrics should take memory performance into account, counting load and stores, not just time elapsed or instruction counts. Null operation lock/unlock microbenchmarks are not reflective of real-world locking: they should incorporate some interim operation.
!!!!Comments & Questions
*A short position paper (no original research), but cited
*Did the gap between processing speed and memory speed widen?
*Beyond time metrics: counting different primitives (load /store).
!!![<<cite Chen1996>>] on performance of commodity OSes
This research paper compares three commodity operating systems. They measure performance at the level of benchmarks as well as individual applications with representative workloads. The commodity systems (Windows for Workgroups, Windows NT, and ~NetBSD Unix) are comparable because they are widely available and support the same uses (although the systems differ substantially at their system-structure level, with Windows lacking protected address spaces, preemptive multitasking, high-level system abstractions such as pipes and background jobs; Windows NT has these). They argue that published benchmarks are misleading or incomplete, and often for commercial purposes.
Their methodology was that of gathering stats for the microbenchmarks, explaining and designing how they would be used. Then, statistics were gathered for the applications, which were selected based on microbenchmark results. They relied on event and cycle counters that were accessible via device driver-level extensions (Table 2 summarizes which low-level events were logged: data reads/writes, TLB misses, data read/write cache misses, code TLB misses, code cache misses, segment register loads, instruction executed, write stall cyles, hardware interrupts).
Their microbenchmark suite was largely based on <<cite McVoy1996>>'s //lmbench// suite, with the addition of a few new measurements, which served as the basis for the application-level tests:
*//null//: latency to access the counter (baseline, not in //lmbench//), useful for counter maintenance and measuring overhead of the measurement itself
*//syscall//: latency of a system call, indication of the cost of accessing functionality implemented in the system; invocation is a small percentage of the total latency
*//exec//: running a trivial program (avg. over 50 invocations)
*//memory access time//: referencing a large array, stride of 128 bytes: results show that OS can affect performance in other ways aside from latency
*//fs//: file system tests: operations that hit in the disk cache, access to small files that go to disk, and file creation: realizing that actual use will require a blend of these operations. Results show a significant effect of system structure
*//bitblt//: a graphics benchmark (not from //lmbench//): allowed them to gain insight on the //ghostscript// workload (see below), other aspects of graphics performance.
*//netbw//: network throughput, informed //WWW server// application load (see below)Their applications were available on all 3 platforms, each had a working set under 32MB, no significant paging:
*//wish//: tcl-tk command interpreter, CPU intensive and windowing system, many context switches between application and graphics server, high instruction count dominates good cache performance
*//ghostscript//: postscript viewer: heavy use of the windowing system, dynamically linked, informed by //bitblt// microbenchmark
*//WWW server//: heavy use of the the network and file systemThey limited background activity in the system, disabled the network (except in the tests that required network access), and used single-user mode. They measured instruction and cycle counts and number of data read references and write references for each experiment.
>//the relevance of microbenchmarks to the performance of realistic workloads is limited//
!!!!Comments & Questions
*They didn't have the source code for Windows, Windows NT (they had a debugger)
*Finding applications that ran on all 3 platforms a struggle (open source, research applications, not "shrink-wrapped" applications), affects realism - likely biased toward ~NetBSD, since applications were originally developed for UNIX; workloads are UNIX centric
*Structural differences and instruction format differences (e. g. multicycle instructions) limited what could be compares at the level of low-level system behaviour; they couldn't discern/compare events logged at different protection levels (user/kernel); idle time was measured differently; they had to resolve timing measurements and which metrics could be compared; instruction counts harder to compare than cycle counts. They couldn't assign cycles to specific events. Policies of the Pentium cache also interfered with data collection.
*how useful (externally valid) is the //exec// microbenchmark? What is a trivial program?
*limited background activity in the system, disabled the network, and used single-user mode: this also affects realism; shouldn't they have used this as a baseline and compared against a realistic setting where these features were all in use?
*dynamic linking in applications made it difficult to measure memory footprint
*did not find evidence to support unified address space model (as in Exokernels)
*performance penalties associated with Windows' backward compatibility (system call hooks)
*they throw up their hands and admit that microbenchmarks are more useful because they're easy to analyze, and they claim that the behaviour they isolated in the microbenchmarks has a significant impact in realistic situations. But they may not be representative! They couldn't find agreed upon macro-metrics in the application workload evaluation.
*What happened to the gathering stats part of their methodology? Stats didn't inform their use of //lmbench//, they just used it because it was readily available. This is prior to criticism of //lmbench// and //hbench:OS// (<<cite Brown1997>>), prior to trace-based methodologies
!!![<<cite Traeger2008>>] file system / storage benchmarking
Traegar and colleagues present a survey of nine years' worth of file systems and storage research papers from top tier venues (OSDI, SOSP, USENIX); in total, 106 papers published between 1999 and 2007 were included in the survey. Specifically, they examined and compares methods and methodologies for performing performance evaluations and the reasoning behind the use of benchmarks. The article is prescriptive in addition to being descriptive, in that it both describes previous work while also discussing their shortcomings and strengths, while also providing guidance for future research; this includes the creation and use of benchmarks, as well as how to present results. Much of this guidance may be generally applicable to performance evaluation in other areas of systems research, however the authors mention that evaluating file and storage systems often requires extra care, in that these systems may interact in complex ways with other subsystems, and may differ from other subsystems in terms of their underlying media, their storage environment, their expected workloads, and their features.
Above all, when evaluating a file or storage system, the authors use to explain what was done in as much detail as possible, and also to explain why it was done that way. As for the former, the state of the system should be reported: is the cache warm or cold? For disk storage benchmarks, where are the partitions located? Is the system empty, or has the system been in real-world use, and if so, for how long? Are other nonessential services running, and if so, what interactions occur? Are workloads multithreaded? When running the evaluations, care should be taken such that each test run is identical, standard deviations and confidence intervals should be computed in the same way. Automated scripts may perform these tasks, thereby limiting human error. When presenting the results, confidence intervals are recommended over standard deviation, which produces a better sense of the true mean, generally decreasing as more runs are performed; the standard deviation, capturing the variation between runs, may not decrease. A normal distribution can only be assumed for more than 30 runs, smaller than this is considered to be a small sample size, where a student-t distribution is more appropriate. Anomalous results, large confidence intervals or non-normal distributions should not be discarded; a software bug or erroneous benchmarking script may be the cause. Regardless of the result, reporting an evaluation in detail will aid in reproducibility; a research group may need to revisit their own prior methodology, or they may want to validate or refute another group's claim.
The purpose of the evaluation should be clear. It may be to compare against other similar systems, to examine the performance when subjected to an expected workload, or to identify the causes of performance overheads or improvements. Of these, the first is most meaningful to readers, so it is often worthwhile to include a comparison to another system in an evaluation. The latter often requires testing several configurations in turn. It is also encouraged to evaluate both high-level and low-level performance, usually satisfied by either macrobenchmarks or a trace-based evaluation (high level) and a set of microbenchmarks (low level). It is important to consider the realism of the both macrobenchmarks and traces. Low-level performance is more meaningful when used to explain high-level performance differences, highlighting worse-case behaviour, isolating specific effects or interactions. In any case, accuracy and scalability should be considered along with realism.
In their survey, Traegar and colleagues discovered that many of the aforementioned details are not reported in the majority of the 106 papers surveyed; others were partially reported. This included the number of benchmark runs, the statistical dispersion, benchmark runtimes, the number of benchmarks, and the system description and state of the cache.
Next, they survey the types of evaluations reported in the survey, discussing trends and common approaches: macrobenchmarks, trace-based evaluations, and microbenchmarks.
Macrobenchmarks aim to simulate a real-world workload, however these depend largely on context, and many papers fail to describe the reasoning for choosing to use a macrobenchmark. An example is the Postmark benchmark, which uses a configurable synthetic workload (but doesn't perform any actual application processing itself). The size is not configurable, so it doesn't scale to modern systems (as of 2008). Another inefficiency is Postmark's file selection algorithm, which is O(N) on the number of files. The authors suggest that a configurable and accurately measurable run time would be more scalable than a configurable workload size. Compile macrobenchmarks are another flavour, however they vary considerably from machine to machine and between different compiler chains. The authors empirically debunk the assumption that compile file systems see similar loads, independent of the software being compiled; in reality these vary from run to run on the same machine. Other popular macrobenchmarks, such as the Andrew File System and NetNews benchmarks, suffer from from being outdated, not reflective of modern applications, and/or do not take into account modern cache behaviour. Commercial benchmarks, such as TPC, SPC, and SPEC benchmarks, are not widely or freely available, often not run according to their specifications, and do not provide configurable operation mixes.
Replayable traces are another approach to system evaluation. While the recorded trace is a real-world workload, questions of generalizability or realism must still be asked. Some papers recorded traces of macrobenchmarks, which is as questionable as the the macrobenchmarks themselves, and may defeat the purpose of a trace (though some macrobenchmarks are proprietary, as described above). There is no clearly-defined way to record and replay traces. Traces can be recorded at several levels, from the level of system calls to that of network or driver protocols, requiring a compromise between level of detail and portability. Trace recording tools are seldom made available or reported in sufficient detail. Trace replaying is likely to occur at the level it was recorded, and may involve aging a file system, and this process should be explained. Replay speed should be justified: should the trace be run at the speed it was recorded, or as fast as possible? Finally, traces should be made available for promoting reproducible research, and yet precautions must be taken to ensure the anonymity of the trace data.
Microbenchmarks test a small number of low-level operations, highlighting performance overheads or benefits implied by macrobenchmark or trace results, or to isolate a specific aspect of the system. Popular microbenchmarks share many of the same considerations described above for macrobenchmarks: scalability, cache effects, generalizability, portability, configurability, and accuracy. Some microbenchmarks are trivially simple to reproduce, even if not publicly or freely available, and yet variations in implementations can lead to significant differences in results, as shown by the authors. Ad hoc, single-use microbenchmarks, are the most difficult to reproduce and are most prone to bugs, however these may be used in conjunction with other microbenchmarks, such as for explaining some unexpected or anomalous result. They may also be useful in the initial phases of benchmarking, to explore the behaviour of the system, and can guide the selection of more widely accepted and available benchmarks. Another form of microbenchmarking is the use of standard system utilities to create a workload, such as wc, grep, cp, diff, tar, and gzip. These utilities are widely available and understood; however, versions of these utilities are subject to change, and they may not scale, given different input files.
The authors also review several workload generator suites, which are widely used and can generate reproducible workload, which require less effort to configure or design a benchmarking procedure. These tools allow benchmarks to be run with exactly the same workload. However, they range in terms of their flexibility / customization.
Their own benchmarking methodology was to run each test at least 10 times, computer 95% confidence intervals mean elapsed, system, and user times using the student-t distribution. They disabled unrelated system services and rebooted the machine between successive sequences of benchmark runs, ensuring consistent cache states. They automated benchmark runs using the Autopilot benchmarking suite, which was a prior project of one of the authors. They demonstrate that different benchmarking decisions can hide overheads and latencies, particularly with regards to compile time benchmarks.
Their findings are summarized in their conclusion, which highlight the need for a method to compare multiple workloads, to gauge their similarity, as well as methods for normalizing results for the machine on which they were run.
!!!!Comments & Questions
*How much can be generalized to other subsystems? To broader OS research?
*Odd structure of the paper; ﮠbenchmarking methodology refers to their own, but then they proceed to examine types of benchmarks, only returning to their own experiments in *configurable workload generators (頡re difficult to tease apart from other benchmarking suites. Iometer, for instance, can be used to specify how long a test should run, rather than the amount of work to be performed. This division seems odd to me.
!!References
<<bibliography Bibliography-CPSC508-Project showAll>>
/***
|Name|CalendarPlugin|
|Source|http://www.TiddlyTools.com/#CalendarPlugin|
|Version|1.5.1|
|Author|Eric Shulman|
|Original Author|SteveRumsby|
|License|unknown|
|~CoreVersion|2.1|
|Type|plugin|
|Description|display monthly and yearly calendars|
NOTE: For //enhanced// date popup display, optionally install:
*[[DatePlugin]]
*[[ReminderMacros|http://remindermacros.tiddlyspot.com/]]
!!!Usage:
<<<
|{{{<<calendar>>}}}|full-year calendar for the current year|
|{{{<<calendar year>>}}}|full-year calendar for the specified year|
|{{{<<calendar year month>>}}}|one month calendar for the specified month and year|
|{{{<<calendar thismonth>>}}}|one month calendar for the current month|
|{{{<<calendar lastmonth>>}}}|one month calendar for last month|
|{{{<<calendar nextmonth>>}}}|one month calendar for next month|
|{{{<<calendar +n>>}}}<br>{{{<<calendar -n>>}}}|one month calendar for a month +/- 'n' months from now|
<<<
!!!Configuration:
<<<
|''First day of week:''<br>{{{config.options.txtCalFirstDay}}}|<<option txtCalFirstDay>>|(Monday = 0, Sunday = 6)|
|''First day of weekend:''<br>{{{config.options.txtCalStartOfWeekend}}}|<<option txtCalStartOfWeekend>>|(Monday = 0, Sunday = 6)|
<<option chkDisplayWeekNumbers>> Display week numbers //(note: Monday will be used as the start of the week)//
|''Week number display format:''<br>{{{config.options.txtWeekNumberDisplayFormat }}}|<<option txtWeekNumberDisplayFormat >>|
|''Week number link format:''<br>{{{config.options.txtWeekNumberLinkFormat }}}|<<option txtWeekNumberLinkFormat >>|
<<<
!!!Revisions
<<<
2011.01.04 1.5.1 corrected parameter handling for {{{<<calendar year>>}}} to show entire year instead of just first month. In createCalendarMonthHeader(), fixed next/previous month year calculation (use parseInt() to convert to numeric value). Code reduction (setting options).
2009.04.31 1.5.0 rewrote onClickCalendarDate() (popup handler) and added config.options.txtCalendarReminderTags. Partial code reduction/cleanup. Assigned true version number (1.5.0)
2008.09.10 added '+n' (and '-n') param to permit display of relative months (e.g., '+6' means 'six months from now', '-3' means 'three months ago'. Based on suggestion from Jean.
2008.06.17 added support for config.macros.calendar.todaybg
2008.02.27 in handler(), DON'T set hard-coded default date format, so that *customized* value (pre-defined in config.macros.calendar.journalDateFmt is used.
2008.02.17 in createCalendarYear(), fix next/previous year calculation (use parseInt() to convert to numeric value). Also, use journalDateFmt for date linking when NOT using [[DatePlugin]].
2008.02.16 in createCalendarDay(), week numbers now created as TiddlyLinks, allowing quick creation/navigation to 'weekly' journals (based on request from Kashgarinn)
2008.01.08 in createCalendarMonthHeader(), 'month year' heading is now created as TiddlyLink, allowing quick creation/navigation to 'month-at-a-time' journals
2007.11.30 added 'return false' to onclick handlers (prevent IE from opening blank pages)
2006.08.23 added handling for weeknumbers (code supplied by Martin Budden (see 'wn**' comment marks). Also, incorporated updated by Jeremy Sheeley to add caching for reminders (see [[ReminderMacros]], if installed)
2005.10.30 in config.macros.calendar.handler(), use 'tbody' element for IE compatibility. Also, fix year calculation for IE's getYear() function (which returns '2005' instead of '105'). Also, in createCalendarDays(), use showDate() function (see [[DatePlugin]], if installed) to render autostyled date with linked popup. Updated calendar stylesheet definition: use .calendar class-specific selectors, add text centering and margin settings
2006.05.29 added journalDateFmt handling
<<<
!!!Code
***/
//{{{
version.extensions.CalendarPlugin= { major: 1, minor: 5, revision: 1, date: new Date(2011,1,4)};
// COOKIE OPTIONS
var opts={
txtCalFirstDay: 0,
txtCalStartOfWeekend: 5,
chkDisplayWeekNumbers: false,
txtCalFirstDay: 0,
txtWeekNumberDisplayFormat: 'w0WW',
txtWeekNumberLinkFormat: 'YYYY-w0WW',
txtCalendarReminderTags: 'reminder'
};
for (var id in opts) if (config.options[id]===undefined) config.options[id]=opts[id];
// INTERNAL CONFIGURATION
config.macros.calendar = {
monthnames:['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'],
daynames:['M','T','W','T','F','S','S'],
todaybg:'#ccccff',
weekendbg:'#c0c0c0',
monthbg:'#e0e0e0',
holidaybg:'#ffc0c0',
journalDateFmt:'DD MMM YYYY',
monthdays:[31,28,31,30,31,30,31,31,30,31,30,31],
holidays:[ ] // for customization see [[CalendarPluginConfig]]
};
//}}}
//{{{
function calendarIsHoliday(date)
{
var longHoliday = date.formatString('0DD/0MM/YYYY');
var shortHoliday = date.formatString('0DD/0MM');
for(var i = 0; i < config.macros.calendar.holidays.length; i++) {
if( config.macros.calendar.holidays[i]==longHoliday
|| config.macros.calendar.holidays[i]==shortHoliday)
return true;
}
return false;
}
//}}}
//{{{
config.macros.calendar.handler = function(place,macroName,params) {
var calendar = createTiddlyElement(place, 'table', null, 'calendar', null);
var tbody = createTiddlyElement(calendar, 'tbody');
var today = new Date();
var year = today.getYear();
if (year<1900) year+=1900;
// get journal format from SideBarOptions (ELS 5/29/06 - suggested by MartinBudden)
var text = store.getTiddlerText('SideBarOptions');
var re = new RegExp('<<(?:newJournal)([^>]*)>>','mg'); var fm = re.exec(text);
if (fm && fm[1]!=null) { var pa=fm[1].readMacroParams(); if (pa[0]) this.journalDateFmt = pa[0]; }
var month=-1;
if (params[0] == 'thismonth') {
var month=today.getMonth();
} else if (params[0] == 'lastmonth') {
var month = today.getMonth()-1; if (month==-1) { month=11; year--; }
} else if (params[0] == 'nextmonth') {
var month = today.getMonth()+1; if (month>11) { month=0; year++; }
} else if (params[0]&&'+-'.indexOf(params[0].substr(0,1))!=-1) {
var month = today.getMonth()+parseInt(params[0]);
if (month>11) { year+=Math.floor(month/12); month%=12; };
if (month<0) { year+=Math.floor(month/12); month=12+month%12; }
} else if (params[0]) {
year = params[0];
if(params[1]) {
month=parseInt(params[1])-1;
if (month>11) month=11; if (month<0) month=0;
}
}
if (month!=-1) {
cacheReminders(new Date(year, month, 1, 0, 0), 31);
createCalendarOneMonth(tbody, year, month);
} else {
cacheReminders(new Date(year, 0, 1, 0, 0), 366);
createCalendarYear(tbody, year);
}
window.reminderCacheForCalendar = null;
}
//}}}
//{{{
// cache used to store reminders while the calendar is being rendered
// it will be renulled after the calendar is fully rendered.
window.reminderCacheForCalendar = null;
//}}}
//{{{
function cacheReminders(date, leadtime)
{
if (window.findTiddlersWithReminders == null) return;
window.reminderCacheForCalendar = {};
var leadtimeHash = [];
leadtimeHash [0] = 0;
leadtimeHash [1] = leadtime;
var t = findTiddlersWithReminders(date, leadtimeHash, null, 1);
for(var i = 0; i < t.length; i++) {
//just tag it in the cache, so that when we're drawing days, we can bold this one.
window.reminderCacheForCalendar[t[i]['matchedDate']] = 'reminder:' + t[i]['params']['title'];
}
}
//}}}
//{{{
function createCalendarOneMonth(calendar, year, mon)
{
var row = createTiddlyElement(calendar, 'tr');
createCalendarMonthHeader(calendar, row, config.macros.calendar.monthnames[mon]+' '+year, true, year, mon);
row = createTiddlyElement(calendar, 'tr');
createCalendarDayHeader(row, 1);
createCalendarDayRowsSingle(calendar, year, mon);
}
//}}}
//{{{
function createCalendarMonth(calendar, year, mon)
{
var row = createTiddlyElement(calendar, 'tr');
createCalendarMonthHeader(calendar, row, config.macros.calendar.monthnames[mon]+' '+ year, false, year, mon);
row = createTiddlyElement(calendar, 'tr');
createCalendarDayHeader(row, 1);
createCalendarDayRowsSingle(calendar, year, mon);
}
//}}}
//{{{
function createCalendarYear(calendar, year)
{
var row;
row = createTiddlyElement(calendar, 'tr');
var back = createTiddlyElement(row, 'td');
var backHandler = function() {
removeChildren(calendar);
createCalendarYear(calendar, parseInt(year)-1);
return false; // consume click
};
createTiddlyButton(back, '<', 'Previous year', backHandler);
back.align = 'center';
var yearHeader = createTiddlyElement(row, 'td', null, 'calendarYear', year);
yearHeader.align = 'center';
yearHeader.setAttribute('colSpan',config.options.chkDisplayWeekNumbers?22:19);//wn**
var fwd = createTiddlyElement(row, 'td');
var fwdHandler = function() {
removeChildren(calendar);
createCalendarYear(calendar, parseInt(year)+1);
return false; // consume click
};
createTiddlyButton(fwd, '>', 'Next year', fwdHandler);
fwd.align = 'center';
createCalendarMonthRow(calendar, year, 0);
createCalendarMonthRow(calendar, year, 3);
createCalendarMonthRow(calendar, year, 6);
createCalendarMonthRow(calendar, year, 9);
}
//}}}
//{{{
function createCalendarMonthRow(cal, year, mon)
{
var row = createTiddlyElement(cal, 'tr');
createCalendarMonthHeader(cal, row, config.macros.calendar.monthnames[mon], false, year, mon);
createCalendarMonthHeader(cal, row, config.macros.calendar.monthnames[mon+1], false, year, mon);
createCalendarMonthHeader(cal, row, config.macros.calendar.monthnames[mon+2], false, year, mon);
row = createTiddlyElement(cal, 'tr');
createCalendarDayHeader(row, 3);
createCalendarDayRows(cal, year, mon);
}
//}}}
//{{{
function createCalendarMonthHeader(cal, row, name, nav, year, mon)
{
var month;
if (nav) {
var back = createTiddlyElement(row, 'td');
back.align = 'center';
back.style.background = config.macros.calendar.monthbg;
var backMonHandler = function() {
var newyear = year;
var newmon = mon-1;
if(newmon == -1) { newmon = 11; newyear = parseInt(newyear)-1;}
removeChildren(cal);
cacheReminders(new Date(newyear, newmon , 1, 0, 0), 31);
createCalendarOneMonth(cal, newyear, newmon);
return false; // consume click
};
createTiddlyButton(back, '<', 'Previous month', backMonHandler);
month = createTiddlyElement(row, 'td', null, 'calendarMonthname')
createTiddlyLink(month,name,true);
month.setAttribute('colSpan', config.options.chkDisplayWeekNumbers?6:5);//wn**
var fwd = createTiddlyElement(row, 'td');
fwd.align = 'center';
fwd.style.background = config.macros.calendar.monthbg;
var fwdMonHandler = function() {
var newyear = year;
var newmon = mon+1;
if(newmon == 12) { newmon = 0; newyear = parseInt(newyear)+1;}
removeChildren(cal);
cacheReminders(new Date(newyear, newmon , 1, 0, 0), 31);
createCalendarOneMonth(cal, newyear, newmon);
return false; // consume click
};
createTiddlyButton(fwd, '>', 'Next month', fwdMonHandler);
} else {
month = createTiddlyElement(row, 'td', null, 'calendarMonthname', name)
month.setAttribute('colSpan',config.options.chkDisplayWeekNumbers?8:7);//wn**
}
month.align = 'center';
month.style.background = config.macros.calendar.monthbg;
}
//}}}
//{{{
function createCalendarDayHeader(row, num)
{
var cell;
for(var i = 0; i < num; i++) {
if (config.options.chkDisplayWeekNumbers) createTiddlyElement(row, 'td');//wn**
for(var j = 0; j < 7; j++) {
var d = j + (config.options.txtCalFirstDay - 0);
if(d > 6) d = d - 7;
cell = createTiddlyElement(row, 'td', null, null, config.macros.calendar.daynames[d]);
if(d == (config.options.txtCalStartOfWeekend-0) || d == (config.options.txtCalStartOfWeekend-0+1))
cell.style.background = config.macros.calendar.weekendbg;
}
}
}
//}}}
//{{{
function createCalendarDays(row, col, first, max, year, mon) {
var i;
if (config.options.chkDisplayWeekNumbers){
if (first<=max) {
var ww = new Date(year,mon,first);
var td=createTiddlyElement(row, 'td');//wn**
var link=createTiddlyLink(td,ww.formatString(config.options.txtWeekNumberLinkFormat),false);
link.appendChild(document.createTextNode(
ww.formatString(config.options.txtWeekNumberDisplayFormat)));
}
else createTiddlyElement(row, 'td');//wn**
}
for(i = 0; i < col; i++)
createTiddlyElement(row, 'td');
var day = first;
for(i = col; i < 7; i++) {
var d = i + (config.options.txtCalFirstDay - 0);
if(d > 6) d = d - 7;
var daycell = createTiddlyElement(row, 'td');
var isaWeekend=((d==(config.options.txtCalStartOfWeekend-0)
|| d==(config.options.txtCalStartOfWeekend-0+1))?true:false);
if(day > 0 && day <= max) {
var celldate = new Date(year, mon, day);
// ELS 10/30/05 - use <<date>> macro's showDate() function to create popup
// ELS 05/29/06 - use journalDateFmt
if (window.showDate) showDate(daycell,celldate,'popup','DD',
config.macros.calendar.journalDateFmt,true, isaWeekend);
else {
if(isaWeekend) daycell.style.background = config.macros.calendar.weekendbg;
var title = celldate.formatString(config.macros.calendar.journalDateFmt);
if(calendarIsHoliday(celldate))
daycell.style.background = config.macros.calendar.holidaybg;
var now=new Date();
if ((now-celldate>=0) && (now-celldate<86400000)) // is today?
daycell.style.background = config.macros.calendar.todaybg;
if(window.findTiddlersWithReminders == null) {
var link = createTiddlyLink(daycell, title, false);
link.appendChild(document.createTextNode(day));
} else
var button = createTiddlyButton(daycell, day, title, onClickCalendarDate);
}
}
day++;
}
}
//}}}
//{{{
// Create a pop-up containing:
// * a link to a tiddler for this date
// * a 'new tiddler' link to add a reminder for this date
// * links to current reminders for this date
// NOTE: this code is only used if [[ReminderMacros]] is installed AND [[DatePlugin]] is //not// installed.
function onClickCalendarDate(ev) { ev=ev||window.event;
var d=new Date(this.getAttribute('title')); var date=d.formatString(config.macros.calendar.journalDateFmt);
var p=Popup.create(this); if (!p) return;
createTiddlyLink(createTiddlyElement(p,'li'),date,true);
var rem='\\n\\<\\<reminder day:%0 month:%1 year:%2 title: \\>\\>';
rem=rem.format([d.getDate(),d.getMonth()+1,d.getYear()+1900]);
var cmd="<<newTiddler label:[[new reminder...]] prompt:[[add a new reminder to '%0']]"
+" title:[[%0]] text:{{store.getTiddlerText('%0','')+'%1'}} tag:%2>>";
wikify(cmd.format([date,rem,config.options.txtCalendarReminderTags]),p);
createTiddlyElement(p,'hr');
var t=findTiddlersWithReminders(d,[0,31],null,1);
for(var i=0; i<t.length; i++) {
var link=createTiddlyLink(createTiddlyElement(p,'li'), t[i].tiddler, false);
link.appendChild(document.createTextNode(t[i]['params']['title']));
}
Popup.show(); ev.cancelBubble=true; if (ev.stopPropagation) ev.stopPropagation(); return false;
}
//}}}
//{{{
function calendarMaxDays(year, mon)
{
var max = config.macros.calendar.monthdays[mon];
if(mon == 1 && (year % 4) == 0 && ((year % 100) != 0 || (year % 400) == 0)) max++;
return max;
}
//}}}
//{{{
function createCalendarDayRows(cal, year, mon)
{
var row = createTiddlyElement(cal, 'tr');
var first1 = (new Date(year, mon, 1)).getDay() -1 - (config.options.txtCalFirstDay-0);
if(first1 < 0) first1 = first1 + 7;
var day1 = -first1 + 1;
var first2 = (new Date(year, mon+1, 1)).getDay() -1 - (config.options.txtCalFirstDay-0);
if(first2 < 0) first2 = first2 + 7;
var day2 = -first2 + 1;
var first3 = (new Date(year, mon+2, 1)).getDay() -1 - (config.options.txtCalFirstDay-0);
if(first3 < 0) first3 = first3 + 7;
var day3 = -first3 + 1;
var max1 = calendarMaxDays(year, mon);
var max2 = calendarMaxDays(year, mon+1);
var max3 = calendarMaxDays(year, mon+2);
while(day1 <= max1 || day2 <= max2 || day3 <= max3) {
row = createTiddlyElement(cal, 'tr');
createCalendarDays(row, 0, day1, max1, year, mon); day1 += 7;
createCalendarDays(row, 0, day2, max2, year, mon+1); day2 += 7;
createCalendarDays(row, 0, day3, max3, year, mon+2); day3 += 7;
}
}
//}}}
//{{{
function createCalendarDayRowsSingle(cal, year, mon)
{
var row = createTiddlyElement(cal, 'tr');
var first1 = (new Date(year, mon, 1)).getDay() -1 - (config.options.txtCalFirstDay-0);
if(first1 < 0) first1 = first1+ 7;
var day1 = -first1 + 1;
var max1 = calendarMaxDays(year, mon);
while(day1 <= max1) {
row = createTiddlyElement(cal, 'tr');
createCalendarDays(row, 0, day1, max1, year, mon); day1 += 7;
}
}
//}}}
//{{{
setStylesheet('.calendar, .calendar table, .calendar th, .calendar tr, .calendar td { text-align:center; } .calendar, .calendar a { margin:0px !important; padding:0px !important; }', 'calendarStyles');
//}}}
/***
|Name|CheckboxPlugin|
|Source|http://www.TiddlyTools.com/#CheckboxPlugin|
|Documentation|http://www.TiddlyTools.com/#CheckboxPluginInfo|
|Version|2.4.0|
|Author|Eric Shulman|
|License|http://www.TiddlyTools.com/#LegalStatements|
|~CoreVersion|2.1|
|Type|plugin|
|Description|Add checkboxes to your tiddler content|
This plugin extends the TiddlyWiki syntax to allow definition of checkboxes that can be embedded directly in tiddler content. Checkbox states are preserved by:
* by setting/removing tags on specified tiddlers,
* or, by setting custom field values on specified tiddlers,
* or, by saving to a locally-stored cookie ID,
* or, automatically modifying the tiddler content (deprecated)
When an ID is assigned to the checkbox, it enables direct programmatic access to the checkbox DOM element, as well as creating an entry in TiddlyWiki's config.options[ID] internal data. In addition to tracking the checkbox state, you can also specify custom javascript for programmatic initialization and onClick event handling for any checkbox, so you can provide specialized side-effects in response to state changes.
!!!!!Documentation
>see [[CheckboxPluginInfo]]
!!!!!Revisions
<<<
2008.01.08 [*.*.*] plugin size reduction: documentation moved to [[CheckboxPluginInfo]]
2008.01.05 [2.4.0] set global "window.place" to current checkbox element when processing checkbox clicks. This allows init/beforeClick/afterClick handlers to reference RELATIVE elements, including using "story.findContainingTiddler(place)". Also, wrap handlers in "function()" so "return" can be used within handler code.
|please see [[CheckboxPluginInfo]] for additional revision details|
2005.12.07 [0.9.0] initial BETA release
<<<
!!!!!Code
***/
//{{{
version.extensions.CheckboxPlugin = {major: 2, minor: 4, revision:0 , date: new Date(2008,1,5)};
//}}}
//{{{
config.checkbox = { refresh: { tagged:true, tagging:true, container:true } };
config.formatters.push( {
name: "checkbox",
match: "\\[[xX_ ][\\]\\=\\(\\{]",
lookahead: "\\[([xX_ ])(=[^\\s\\(\\]{]+)?(\\([^\\)]*\\))?({[^}]*})?({[^}]*})?({[^}]*})?\\]",
handler: function(w) {
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
// get params
var checked=(lookaheadMatch[1].toUpperCase()=="X");
var id=lookaheadMatch[2];
var target=lookaheadMatch[3];
if (target) target=target.substr(1,target.length-2).trim(); // trim off parentheses
var fn_init=lookaheadMatch[4];
var fn_clickBefore=lookaheadMatch[5];
var fn_clickAfter=lookaheadMatch[6];
var tid=story.findContainingTiddler(w.output); if (tid) tid=tid.getAttribute("tiddler");
var srctid=w.tiddler?w.tiddler.title:null;
config.macros.checkbox.create(w.output,tid,srctid,w.matchStart+1,checked,id,target,config.checkbox.refresh,fn_init,fn_clickBefore,fn_clickAfter);
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
}
} );
config.macros.checkbox = {
handler: function(place,macroName,params,wikifier,paramString,tiddler) {
if(!(tiddler instanceof Tiddler)) { // if no tiddler passed in try to find one
var here=story.findContainingTiddler(place);
if (here) tiddler=store.getTiddler(here.getAttribute("tiddler"))
}
var srcpos=0; // "inline X" not applicable to macro syntax
var target=params.shift(); if (!target) target="";
var defaultState=params[0]=="checked"; if (defaultState) params.shift();
var id=params.shift(); if (id && !id.length) id=null;
var fn_init=params.shift(); if (fn_init && !fn_init.length) fn_init=null;
var fn_clickBefore=params.shift();
if (fn_clickBefore && !fn_clickBefore.length) fn_clickBefore=null;
var fn_clickAfter=params.shift();
if (fn_clickAfter && !fn_clickAfter.length) fn_clickAfter=null;
var refresh={ tagged:true, tagging:true, container:false };
this.create(place,tiddler.title,tiddler.title,0,defaultState,id,target,refresh,fn_init,fn_clickBefore,fn_clickAfter);
},
create: function(place,tid,srctid,srcpos,defaultState,id,target,refresh,fn_init,fn_clickBefore,fn_clickAfter) {
// create checkbox element
var c = document.createElement("input");
c.setAttribute("type","checkbox");
c.onclick=this.onClickCheckbox;
c.srctid=srctid; // remember source tiddler
c.srcpos=srcpos; // remember location of "X"
c.container=tid; // containing tiddler (may be null if not in a tiddler)
c.tiddler=tid; // default target tiddler
c.refresh = {};
c.refresh.container = refresh.container;
c.refresh.tagged = refresh.tagged;
c.refresh.tagging = refresh.tagging;
place.appendChild(c);
// set default state
c.checked=defaultState;
// track state in config.options.ID
if (id) {
c.id=id.substr(1); // trim off leading "="
if (config.options[c.id]!=undefined)
c.checked=config.options[c.id];
else
config.options[c.id]=c.checked;
}
// track state in (tiddlername|tagname) or (fieldname@tiddlername)
if (target) {
var pos=target.indexOf("@");
if (pos!=-1) {
c.field=pos?target.substr(0,pos):"checked"; // get fieldname (or use default "checked")
c.tiddler=target.substr(pos+1); // get specified tiddler name (if any)
if (!c.tiddler || !c.tiddler.length) c.tiddler=tid; // if tiddler not specified, default == container
if (store.getValue(c.tiddler,c.field)!=undefined)
c.checked=(store.getValue(c.tiddler,c.field)=="true"); // set checkbox from saved state
} else {
var pos=target.indexOf("|"); if (pos==-1) var pos=target.indexOf(":");
c.tag=target;
if (pos==0) c.tag=target.substr(1); // trim leading "|" or ":"
if (pos>0) { c.tiddler=target.substr(0,pos); c.tag=target.substr(pos+1); }
if (!c.tag.length) c.tag="checked";
var t=store.getTiddler(c.tiddler);
if (t && t.tags)
c.checked=t.isTagged(c.tag); // set checkbox from saved state
}
}
// trim off surrounding { and } delimiters from init/click handlers
if (fn_init) c.fn_init="(function(){"+fn_init.trim().substr(1,fn_init.length-2)+"})()";
if (fn_clickBefore) c.fn_clickBefore="(function(){"+fn_clickBefore.trim().substr(1,fn_clickBefore.length-2)+"})()";
if (fn_clickAfter) c.fn_clickAfter="(function(){"+fn_clickAfter.trim().substr(1,fn_clickAfter.length-2)+"})()";
c.init=true; c.onclick(); c.init=false; // compute initial state and save in tiddler/config/cookie
},
onClickCheckbox: function(event) {
window.place=this;
if (this.init && this.fn_init) // custom function hook to set initial state (run only once)
{ try { eval(this.fn_init); } catch(e) { displayMessage("Checkbox init error: "+e.toString()); } }
if (!this.init && this.fn_clickBefore) // custom function hook to override changes in checkbox state
{ try { eval(this.fn_clickBefore) } catch(e) { displayMessage("Checkbox onClickBefore error: "+e.toString()); } }
if (this.id)
// save state in config AND cookie (only when ID starts with 'chk')
{ config.options[this.id]=this.checked; if (this.id.substr(0,3)=="chk") saveOptionCookie(this.id); }
if (this.srctid && this.srcpos>0 && (!this.id || this.id.substr(0,3)!="chk") && !this.tag && !this.field) {
// save state in tiddler content only if not using cookie, tag or field tracking
var t=store.getTiddler(this.srctid); // put X in original source tiddler (if any)
if (t && this.checked!=(t.text.substr(this.srcpos,1).toUpperCase()=="X")) { // if changed
t.set(null,t.text.substr(0,this.srcpos)+(this.checked?"X":"_")+t.text.substr(this.srcpos+1),null,null,t.tags);
if (!story.isDirty(t.title)) story.refreshTiddler(t.title,null,true);
store.setDirty(true);
}
}
if (this.field) {
if (this.checked && !store.tiddlerExists(this.tiddler))
store.saveTiddler(this.tiddler,this.tiddler,"",config.options.txtUserName,new Date());
// set the field value in the target tiddler
store.setValue(this.tiddler,this.field,this.checked?"true":"false");
// DEBUG: displayMessage(this.field+"@"+this.tiddler+" is "+this.checked);
}
if (this.tag) {
if (this.checked && !store.tiddlerExists(this.tiddler))
store.saveTiddler(this.tiddler,this.tiddler,"",config.options.txtUserName,new Date());
var t=store.getTiddler(this.tiddler);
if (t) {
var tagged=(t.tags && t.tags.indexOf(this.tag)!=-1);
if (this.checked && !tagged) { t.tags.push(this.tag); store.setDirty(true); }
if (!this.checked && tagged) { t.tags.splice(t.tags.indexOf(this.tag),1); store.setDirty(true); }
}
// if tag state has been changed, update display of corresponding tiddlers (unless they are in edit mode...)
if (this.checked!=tagged) {
if (this.refresh.tagged) {
if (!story.isDirty(this.tiddler)) // the TAGGED tiddler in view mode
story.refreshTiddler(this.tiddler,null,true);
else // the TAGGED tiddler in edit mode (with tags field)
config.macros.checkbox.refreshEditorTagField(this.tiddler,this.tag,this.checked);
}
if (this.refresh.tagging)
if (!story.isDirty(this.tag)) story.refreshTiddler(this.tag,null,true); // the TAGGING tiddler
}
}
if (!this.init && this.fn_clickAfter) // custom function hook to react to changes in checkbox state
{ try { eval(this.fn_clickAfter) } catch(e) { displayMessage("Checkbox onClickAfter error: "+e.toString()); } }
// refresh containing tiddler (but not during initial rendering, or we get an infinite loop!) (and not when editing container)
if (!this.init && this.refresh.container && this.container!=this.tiddler)
if (!story.isDirty(this.container)) story.refreshTiddler(this.container,null,true); // the tiddler CONTAINING the checkbox
return true;
},
refreshEditorTagField: function(title,tag,set) {
var tagfield=story.getTiddlerField(title,"tags");
if (!tagfield||tagfield.getAttribute("edit")!="tags") return; // if no tags field in editor (i.e., custom template)
var tags=tagfield.value.readBracketedList();
if (tags.contains(tag)==set) return; // if no change needed
if (set) tags.push(tag); // add tag
else tags.splice(tags.indexOf(tag),1); // remove tag
for (var t=0;t<tags.length;t++) tags[t]=String.encodeTiddlyLink(tags[t]);
tagfield.value=tags.join(" "); // reassemble tag string (with brackets as needed)
return;
}
}
//}}}
|Name|CheckboxPluginInfo|
|Source|http://www.TiddlyTools.com/#CheckboxPlugin|
|Documentation|http://www.TiddlyTools.com/#CheckboxPluginInfo|
|Version|2.4.0|
|Author|Eric Shulman|
|License|http://www.TiddlyTools.com/#LegalStatements|
|~CoreVersion|2.1|
|Type|documentation|
|Description|documentation for CheckboxPlugin|
This plugin extends the TiddlyWiki syntax to allow definition of checkboxes that can be embedded directly in tiddler content. Checkbox states are preserved by:
* setting/removing tags on specified tiddlers,
* or, setting custom field values on specified tiddlers,
* or, saving to a locally-stored cookie ID,
* or, automatically modifying the tiddler source content (deprecated).
When an ID is assigned to the checkbox, it enables direct programmatic access to the checkbox DOM element, as well as creating an entry in TiddlyWiki's config.options[ID] internal data. In addition to tracking the checkbox state, you can also specify custom javascript for programmatic initialization and onClick event handling for any checkbox, so you can provide specialized side-effects in response to state changes.
!!!!!Inline (wiki syntax) Usage
<<<
//{{{
[ ]or[_] and [x]or[X]
//}}}
Simple checkboxes using 'Inline X' storage. The current unchecked/checked state is indicated by the character between the {{{[}}} and {{{]}}} brackets ("_" means unchecked, "X" means checked). When you click on a checkbox, the current state is retained by directly modifying the tiddler content to place the corresponding "_" or "X" character in between the brackets.
>//''NOTE: 'Inline X' syntax has been deprecated...'' This storage format only works properly for checkboxes that are directly embedded and accessed from content in a single tiddler. However, if that tiddler is 'transcluded' into another (by using the {{{<<tiddler TiddlerName>>}}} macro), the 'Inline X' will be ''erroneously stored in the containing tiddler's source content, resulting in corrupted content in that tiddler.'' For anything but the most simple of "to do list" uses, you should select from the various alternative storage methods described below...//
//{{{
[x=id]
//}}}
Assign an optional ID to the checkbox so you can use {{{document.getElementByID("id")}}} to manipulate the checkbox DOM element, as well as tracking the current checkbox state in {{{config.options["id"]}}}. If the ID starts with "chk" the checkbox state will also be saved in a cookie, so it can be automatically restored whenever the checkbox is re-rendered (overrides any default {{{[x]}}} or {{{[_]}}} value). If a cookie value is kept, the "_" or "X" character in the tiddler content remains unchanged, and is only applied as the default when a cookie-based value is not currently defined.
//{{{
[x(title|tag)] or [x(title:tag)]
//}}}
Initializes and tracks the current checkbox state by setting or removing a particular tag value from a specified tiddler. If you omit the tiddler title (and the | or : separator), the specified tag is assigned to the current tiddler. If you omit the tag value, as in {{{(title|)}}}, the default tag, {{{checked}}}, is assumed. Omitting both the title and tag, {{{()}}}, tracks the checkbox state by setting the "checked" tag on the current tiddler. When tag tracking is used, the "_" or "X" character in the tiddler content remains unchanged, and is not used to set or track the checkbox state. If a tiddler title named in the tag does not exist, the checkbox state defaults to the "inline X" value. If this value is //checked//, or is subsequently changed to //checked//, it will automatically create the missing tiddler and then add the tag to it. //''NOTE: beginning with version 2.1.2 of this plugin, the "|" separator is the preferred separator between the title and tag name, as it avoids syntactic ambiguity when ":" is used within tiddler titles or tag names.''//
//{{{
[x(field@tiddler)]
//}}}
Initializes and tracks the current checkbox state by setting a particular custom field value from a specified tiddler. If you omit the tiddler title (but not the "@" separator), the specified field on the current tiddler is used. If you omit the field name, as in {{{(@tiddler)}}}, a default fieldname of {{{checked}}} is assumed. Omitting both the field and the tiddler title, {{{(@)}}}, defaults to setting the "checked" field on the current tiddler. When field tracking is used, the "_" or "X" character in the tiddler content remains unchanged, and is not used to set or track the checkbox state. If the tiddler title named in the parameter does not exist, the checkbox state defaults to the "inline X" value. If this value is //checked// or is subsequently changed to //checked//, it will automatically create the missing tiddler and then add the field to it.
//{{{
[x{javascript}{javascript}{javascript}]
//}}}
You can define optional javascript code segments to add custom initialization and/or 'onClick' handlers to a checkbox. The current checkbox state (and it's other DOM attributes) can be set or read from within these code segments by reference to a globally-defined context object, "place" (which can also be referenced as "window.place").
The first code segment will be executed when the checkbox is initially displayed, so that you can programmatically determine it's starting checked/unchecked state. The second code segment (if present) is executed whenever the checkbox is clicked, but //before the regular checkbox processing in performed// ("onClickBefore"), so that you can apply programmed responses or intercept and override the checkbox state based on custom logic. The third code segment (if present) is executed whenver the checkbox is clicked, //after the regular checkbox processing has completed// ("onClickAfter"), so that you can include "side-effect" processing based on the checkbox state just applied.
>Note: if you want to use the default checkbox initialization processing with a custom onClickBefore/After function, use this syntax:
>{{{[x(tag){}{javascript}]}}} or {{{[x(tag){}{}{javascript}]}}}
<<<
!!!!!Macro usage
<<<
In addition to embedded checkboxes using the wiki syntax described above, a ''macro-based syntax'' is also provided, for use in templates where wiki syntax cannot be directly used. This macro syntax can also be used in tiddler content, as an alternative to the wiki syntax. When embedded in [[PageTemplate]], [[ViewTemplate]], or [[EditTemplate]] (or custom alternative templates), use the following macro syntax:
//{{{
<span macro="checkbox target checked id onInit onClickBefore onClickAfter"></span>
//}}}
or, when embedded in tiddler content, use the following macro syntax:
//{{{
<<checkbox target checked id onInit onClickBefore onClickAfter>>
//}}}
where:
''target''
>is either a tag reference (e.g., ''tagname|tiddlername'') or a field reference (e.g. ''fieldname@tiddlername''), as described above.
''checked'' (optional)
>is a keyword that sets the initial state of the checkbox to "checked". When omitted, the default checkbox state is "unchecked".
''id'' (optional)
>specifies an internal config.options.* ID, as described above. If the ID begins with "chk", a cookie-based persistent value will be created to track the checkbox state in between sessions.
''onInit'' (optional)
>contains a javascript event handler to be performed when the checkbox is initially rendered (see details above).
''onClickBefore'' and/or ''onClickAfter'' (optional)
>contains a javascript event handler to be performed each time the checkbox is clicked (see details above). //note: to use the default onInit handler with a custom onClickBefore/After handler, use "" (empty quotes) or {} (empty function) as a placeholder for the onInit and/or onClickBefore parameters//
<<<
!!!!!Examples
<<<
''checked and unchecked static default ("inline X") values:''
//{{{
[X] label
[_] label
//}}}
>[X] label
>[_] label
''document-based value (id='demo', no cookie):''
//{{{
[_=demo] label
//}}}
>[_=demo] label
''cookie-based value (id='chkDemo'):''
//{{{
[_=chkDemo] label
//}}}
>[_=chkDemo] label
''tag-based value (TogglyTagging):''
//{{{
[_(CheckboxPluginInfo|demotag)]
[_(CheckboxPluginInfo|demotag){place.refresh.tagged=place.refresh.container=false}]
//}}}
>[_(CheckboxPluginInfo|demotag)] toggle 'demotag' (and refresh tiddler display)
>[_(CheckboxPluginInfo|demotag){place.refresh.tagged=place.refresh.container=false}] toggle 'demotag' (no refresh)
''field-based values:''
//{{{
[_(demofield@CheckboxPluginInfo)] demofield@CheckboxPluginInfo
[_(demofield@)] demofield@ (equivalent to demonfield@ current tiddler)
[_(checked@CheckboxPluginInfo)] checked@CheckboxPluginInfo
[_(@CheckboxPluginInfo)] @CheckboxPluginInfo
[_(@)] @ (equivalent to checked@ current tiddler)
//}}}
>[_(demofield@CheckboxPluginInfo)] demofield@CheckboxPluginInfo
>[_(demofield@)] demofield@ (current tiddler)
>[_(checked@CheckboxPluginInfo)] checked@CheckboxPluginInfo
>[_(@CheckboxPluginInfo)] @CheckboxPluginInfo
>[_(@)] toggle field: @ (defaults to "checked@here")
>click to view current: <<toolbar fields>>
''custom init and onClick functions:''
//{{{
[X{place.checked=true}{alert(place.checked?"on":"off")}] message box with checkbox state
//}}}
>[X{place.checked=true}{alert(place.checked?"on":"off")}] message box with checkbox state
''retrieving option values:''
>config.options['demo']=<script>return config.options['demo']?"true":"false";</script>
>config.options['chkDemo']=<script>return config.options['chkDemo']?"true":"false";</script>
<<<
!!!!!Configuration
<<<
Normally, when a checkbox state is changed, the affected tiddlers are automatically re-rendered, so that any checkbox-dependent dynamic content can be updated. There are three possible tiddlers to be re-rendered, depending upon where the checkbox is placed, and what kind of storage method it is using.
*''container'': the tiddler in which the checkbox is displayed. (e.g., this tiddler)
*''tagged'': the tiddler that is being tagged (e.g., "~MyTask" when tagging "~MyTask:done")
*''tagging'': the "tag tiddler" (e.g., "~done" when tagging "~MyTask:done")
You can set the default refresh handling for all checkboxes in your document by using the following javascript syntax either in a systemConfig plugin, or as an inline script. (Substitute true/false values as desired):
{{{config.checkbox.refresh = { tagged:true, tagging:true, container:true };}}}
You can also override these defaults for any given checkbox by using an initialization function to set one or more of the refresh options. For example:
{{{[_{place.refresh.container=false}]}}}
<<<
!!!!!Revisions
<<<
2008.01.08 [*.*.*] plugin size reduction: documentation moved to [[CheckboxPluginInfo]]
2008.01.05 2.4.0 set global "window.place" to current checkbox element when processing checkbox clicks. This allows init/beforeClick/afterClick handlers to reference RELATIVE elements, including using "story.findContainingTiddler(place)". Also, wrap handlers in "function()" so "return" can be used within handler code.
2008.01.02 2.3.0 split optional custom onClick handling into separate onClickBefore and onClickAfter handlers. The onClickBefore handler permits interception of the click BEFORE the checkbox is set. onClickAfter allows follow-on 'side-effect' processing to occur AFTER the checkbox is set.
2007.12.04 [*.*.*] update for TW2.3.0: replaced deprecated core functions, regexps, and macros
2007.08.06 2.2.5 supress automatic refresh of any tiddler that is currently being edited. Ensures that current tiddler edit sessions are not prematurely discarded (losing any changes). However, if checkbox changes a tag on a tiddler being edited, update the "tags" input field (if any) so that saving the edited tiddler correctly reflects any changes due to checkbox activity... see refreshEditorTagField().
2007.07.13 - 2.2.4 in handler(), fix srctid reference (was "w.tiddler", should have been "w.tiddler.title"). This fixes broken 'inline X' plus fatal macro error when using PartTiddlerPlugin. Thanks to cmari for reporting the problem and UdoBorkowski for finding the code error.
2007.06.21 - 2.2.3 suppress automatic refresh of tiddler when using macro-syntax to prevent premature end of tiddler editing session.
2007.06.20 - 2.2.2 fixed handling for 'inline X' when checkboxes are contained in a 'trancluded' tiddler. Now, regardless of where an inline X checkbox appears, the X will be placed in the originating source tiddler, rather than the tiddler in which the checkbox appears.
2007.06.17 - 2.2.1 Refactored code to add checkbox //macro// syntax for use in templates (e.g., {{{macro="checkbox ..."}}}. Also, code cleanup of existing tag handling.
2007.06.16 - 2.2.0 added support for tracking checkbox states using tiddler fields via "(fieldname@tiddlername)" syntax.
2006.05.04 - 2.1.3 fix use of findContainingTiddler() to check for a non-null return value, so that checkboxes won't crash when used outside of tiddler display context (such as in header, sidebar or mainmenu)
2006.03.11 - 2.1.2 added "|" as delimiter to tag-based storage syntax (e.g. "tiddler|tag") to avoid parsing ambiguity when tiddler titles or tag names contain ":". Using ":" as a delimiter is still supported but is deprecated in favor of the new "|" usage. Based on a problem reported by JeffMason.
2006.02.25 - 2.1.0 added configuration options to enable/disable forced refresh of tiddlers when toggling tags
2006.02.23 - 2.0.4 when toggling tags, force refresh of the tiddler containing the checkbox.
2006.02.23 - 2.0.3 when toggling tags, force refresh of the 'tagged tiddler' so that tag-related tiddler content (such as "to-do" lists) can be re-rendered.
2006.02.23 - 2.0.2 when using tag-based storage, allow use [[ and ]] to quote tiddler or tag names that contain spaces:
{{{[x([[Tiddler with spaces]]:[[tag with spaces]])]}}}
2006.01.10 - 2.0.1 when toggling tags, force refresh of the 'tagging tiddler'. For example, if you toggle the "systemConfig" tag on a plugin, the corresponding "systemConfig" TIDDLER will be automatically refreshed (if currently displayed), so that the 'tagged' list in that tiddler will remain up-to-date.
2006.01.04 - 2.0.0 update for ~TW2.0
2005.12.27 - 1.1.2 Fix lookAhead regExp handling for {{{[x=id]}}}, which had been including the "]" in the extracted ID.
Added check for "chk" prefix on ID before calling saveOptionCookie()
2005.12.26 - 1.1.2 Corrected use of toUpperCase() in tiddler re-write code when comparing {{{[X]}}} in tiddler content with checkbox state. Fixes a problem where simple checkboxes could be set, but never cleared.
2005.12.26 - 1.1.0 Revise syntax so all optional parameters are included INSIDE the [ and ] brackets. Backward compatibility with older syntax is supported, so content changes are not required when upgrading to the current version of this plugin. Based on a suggestion by GeoffSlocock
2005.12.25 - 1.0.0 added support for tracking checkbox state using tags ("TogglyTagging")
Revised version number for official post-beta release.
2005.12.08 - 0.9.3 support separate 'init' and 'onclick' function definitions.
2005.12.08 - 0.9.2 clean up lookahead pattern
2005.12.07 - 0.9.1 only update tiddler source content if checkbox state is actually different. Eliminates unnecessary tiddler changes (and 'unsaved changes' warnings)
2005.12.07 - 0.9.0 initial BETA release
<<<
CWA is a framework that combines ''constraint-based task analysis'' with ''work domain analysis'' as the foundation for a holistic and socio-technical account of an individual's work involving information technology.
It involves five phases:
#''work domain analysis'' - a descriptive analysis of work practices and unforeseen circumstances, error recovery tasks, tasks with fuzzy goal states
#''control task analysis'' - a device-independent constraint-based task analysis approach based on control theory, allowing for context-dependent disruptions, specifying a goal state and constraints (inputs, outputs, disruptions), a normative analysis of typical tasks
#''strategies analysis'' - a descriptive / formative approach to studying how tasks are performed (steps, techniques, devices)
#''social organization and cooperation analysis''
#''worker competencies analysis''
!!!Normative Work Analysis: Task Analysis
The 3rd chapter of <<cite Vicente1999 bibliography:Bibliography>>'s book outlines approaches to task analysis: ''Task analysis'' has existed since the 1950s and while important part of studying work practices, it alone is insufficient for understanding these work practices or for informing design. Furthermore, most task analysis assumes an ''instruction-based approach to task analysis'', usually a ''flow-based'' or ''sequence-based approach'', in which there is a specific set of steps (or branches of steps) to follow while executing a task (flow- and sequence-based), or a set of durations associated with these steps (sequence-based, having assembly-line precision). Instruction-based approaches are fine for studying "closed-system tasks" and ''device-dependent tasks'', though what is considered to be a truly "closed system" is subjective and likely along a continuum, casting doubt over the true utility of instruction-based approaches. Despite their prevalence, these approaches are particular unsuitable for early design research.
In contrast, the underrepresented ''constraint-based approaches to task analysis'' does not outline steps but constraints on a task, and thus allows for more flexibility and worker discretion in how a task is performed, allowing for context-dependent variability and alternative strategies, such as strategies optimized for speed and strategies optimized for accuracy/precision. Strategies differ between experts and novices, between experts, and between contexts. constraint-based approaches to task analysis is useful for studying "open-system tasks", those which are subject to context-dependent variability and environmental disruption. Tasks are described in a deliberately vague and abstract way to accommodate flexibility in how they are or will eventually be performed. Constraint-based approaches are ''device-independent''; this is preferable since the rationale for performing task analysis is often to improve or introduce a new design for a particular device or set of devices, a design that will change work practices upon its deployment. Rather than outline steps, these approaches place constraints within a large state space on the possibility of some steps and the order in which they might occur, on "what should //not// be done" rather than on "what must be done".
One exemplar constraint-based approach to task analysis is ''input-output'' task analysis, in which a task is specified at a high-level of abstraction according to a goal state along with its input(s) and output(s); //how// the goal state is reached is not part of the analysis, unlike instruction-based approaches to task analysis which outline each step or component of a task.
In constraint-based approaches to task analysis, a ''Strategies Analysis'' phase of CWA complements the task analysis (Rasmussen and Jensen, 1973), documenting //how// tasks are performed adaptively subject to context-dependent variability and disruptions emanating from the environment. Strategies Analysis is intended to be a systematic approach to describing strategies.
In addition, constraint-based task analysis does not answer the question of //who// does the task; as some tasks occur within a socio-technical context, tasks may be automated or performed by a human, or by a team of people working collaboratively. Later phases of CWA, namely ''Social Organization and Cooperation Analysis'' and ''Worker Competencies Analysis'', serve to answer the question of //who//.
All task analysis approaches, regardless of whether they are instruction-based or constraint-based, fall short because they are ''normative perspectives'' on work practices, on "how things should be done": tasks in which the goal state is clearly and objectively known. Task analysis does not accommodate unforeseen circumstances, error recovery tasks, or other anomalies, occasions in which exceptional work practices are carried out, nor does it explain work practices in which the goal state or outputs difficult to specify.
For this reason, task analysis is only part of the CWA framework, and work domain analysis must be performed to better understand work practices in exceptional circumstances, whereas constraint-based task analysis allows practitioners to study typical work practices and clearly defined goal states. The constraint-based task analysis used in CWA is known as ''control task analysis'' (ch. 8), and involves a method called a ''design ladder'', a modeling technique for developing control task models.
!!!Descriptive Work Analysis
While a ''task'' is a prescribed normative action, an ''activity'' is a descriptive account of an action in practice. Several approaches to descriptive work analysis and case study examples are compared in this chapter, from various research communities: Soviet activity theory, human factors, francophone ergonomics, situated action, naturalistic decision making (Klein, 1989 - firefighters case study), distributed cognition (<<cite Hutchins1995>> - navigation case study)
One form of descriptive work analysis is ''Soviet activity theory'''s contribution to HCI (B⬠1991), which separates ends (goals) from means (devices), and imparts upon us the idea that we work //through// computers. not //with// computers.
Another approach is ''link analysis'' (see Kirwan / Ainsworth, 1992), a context- and device-specific approach based on ''activity sampling''. This approach leads designers to fall one step behind their interventions.
Limitations of descriptive work analysis centre around the ''task-artefact'' cycle, in which even scenario-based design and rapid prototyping suffer from device dependence and incompleteness, bound to current practice. The device design should be the output of the process, not the input.
Nevertheless, descriptive approaches are still useful, pointing out existing practices, workarounds, and strategies worth preserving or explicitly designing for. However, descriptive work analysis should be device-independent, and strive to identify ''intrinsic work constraints''.
!!!Formative Work Analysis: Cognitive Work Analysis
CWA is an instance of formative work analysis (Beyer/Holtzblatt's ''contextual design'' is another). The five stages (above) begin with an ecological perspective and culminate in a socio-organizational or cognitivist perspective. CWA combines descriptive (work domain analysis, or how work //is// done currently) and normative (task analysis, or how work //should be// done) into a formative framework (how work //could be// done), which also includes strategies analysis, social/organizational analysis, and worker competencies analysis.
CWA centres around the modeling of intrinsic work constraints along the continuum between ecological and cognitivist perspectives. These contraints are nested, with work domain constraints at the outermost level and worker competencies at the innermost.
In systems designed using a CWA framework, workers "finish the design", in that systems are designed for adaptation. Systems must allow for distributed, rather than centralized control. Workers should not rotely follow system instructions, but be informed of constraints and violations by the system such that they are aware of the possible strategies one could follow to complete the task at hand.
CWA's goals:
#support ''adaptation'' and decision making for unanticipated events
#develop ''useful'' systems that allow for worker discretion in how they are used
#develop ''usable'' systems with worker competencies/limitations and social/organizational structure in mind
#facilitate worker autonomy, allow them to make decisions and gain skills
CWA doesn't necessarily have to be about revolutionary design, it can be part of evolutionary design projects as well, though these will entail more initial constraints, such as on device-dependence and static social-organizational structures; these will serve as inputs rather than formative outputs, which is less than ideal.
!!!Cognitive Work Analysis Phase 1: Work Domain Analysis
This chapter discusses work domain modeling tools, namely the //abstraction decomposition//, which decomposes a work domain into part-whole relationships (abstraction-decomposition space), and an abstraction hierarchy of strucural means-ends relations. The latter involves asking //why//, //what//, and //how// to elucidate these relations between abstract and phsyical entities in a work domain. An example in human factors experimentation is given (fig. 7.10) where the NASA-TLX (what) is a measure of subjective mental workload (why, one level above in the abstraction hierarchy) achieved by a paper form or computer (how, one level below in the abstraction hierarchy). The why-what-how relation is a sliding set, and these can be asked at various levels of the abstraction hierarchy. The level of detail appropriate for analysis is left to the discretion of the analyst and within the constraints of the work domain's part-whole hierarchy. //Structural means-ends relations// concern nouns, and are to be discerned from //action means-ends relations//, which concern verbs and are the subject of control task analysis.
!!!Cognitive Work Analysis Phase 2: Control Task Analysis
Control Task Analysis is a form of constraint-based task analysis that specifies what needs to be done independent of how they are executed (strategies analysis) or who does them (social organization and cooperation analysis). It specifies constraints such as goals and input and output constraints, though it does not adhere to a strict temporal/sequential description of tasks, allowing for context-conditioned variability and expert abilities relating to bypassing interim steps or procedures. Vicente's CTA uses a modeling tool called a //decision ladder// (aka Rasmussen's ''activity analysis''). The decision ladder contains the same ''data processing'' activities described in sequential task analysis (Newell and Simon, Kirwan and Ainsworth, et al.): ''activate'', ''observe'', ''identify'', ''interpret'', ''define task'', ''formulate procedure'', and ''execute''. Between each activity are interim states of knowledge resulting from data processing, serving as input and output to the various activities. The decision ladder is not impose a strictly linear flow between these activities, as some can be bypassed (''shunts'' between activities and states and ''associative leaps'' between states). Furthermore, a task can start at any point in the decision ladder, form cycles, and traverse in either direction. Thus segments of the decision ladder form various subroutines and these can be assembled flexibly to construct task descriptions. The decision ladder is domain- and device-agnostic, serving as a skeleton for domain-specific decision-making activities and information to be filled in. The granularity of description for each activity is subject to the analyst's discretion and to the constraints of the work domain identified in Phase 1. There should be a unique decision ladder for each control task, //prototypical situation//, or //operating mode// in a work domain. The decision ladder captures //action means-ends// relations, rather than //structural means-ends// relations, which should already be accounted for in the WDA phase. An example of a an //action means-ends// relation is that of changing a setting (means) to achieve a desired state (ends).
!!!Cognitive Work Analysis Phase 3: Strategies Analysis
Strategies analysis describes //how// a task is carried out independent of //who// does it, still in a device-agnostic fashion, examining the different possible strategies for each subroutine or activity in the decision ladder (see CTA). Strategies are defined as //categories of cognitive task procedures//, to be distinguished from //instances of cognitive task procedures//, thereby remaining abstract and generalizable to various devices. A hypothetical actor is a stand-in for //who// executes the strategy, as it may be the case that the strategy might be automated or carried out by a team of workers.
Vicente demonstrates the use of a tool called a ''information flow map'' for modeling strategies, though he admits that this tool is still unproven and not at the same level of maturity as the ''decision ladder'' or ''abstraction-decomposition''.
!!!!Comments & Questions
*~InfoVis EDA is an open-system task
**Input-output specification in the multi-level task typology suggests constraint-based task analysis, though our specification of //how// is problematic: this is part of ''strategies analysis'' if done descriptively, and is not typical of task analysis. We could specify that our typology of //how// is descriptive rather than normative.
**//why// and //how// as normative constraint-based task analysis - specifying a goal (why) and input and output (what), //how// as descriptive or formative strategies analysis
*See Kirwan & Ainsworth's 1992 text //A Guide to Task Analysis//, also Kugler et al (1982) as an example of task analysis
*Also see Rasmussen (1979a) on analysis of human errors, and Rasmussen (1979b) on a //morphology (taxonomy?) of mental models in man-machine contexts//
*parallels w/ <<cite Munzner2009>>'s //nested model//: domain problem = work domain; task abstraction = control tasks and strategies; techniques and algorithms = social/organization structure, worker competencies and automation
*Task typology revision cover letter:
>//Work Domain Analysis harmonizes with the early "discover" phase of the design study methodology (Sedlmair et al., 2012), in which a practitioner must understand the constraints of the domain problem and its data, a necessary precursor to the analysis of tasks and strategies for executing these tasks. While WDA is not an immediate focus of our current submission, we will consider approaches to WDA discussed by Vicente in future work building on or incorporating the design study methodology. //
>
>//We noted that Vicente also appreciates the power and simplicity of asking why, what, and how for elucidating means and ends (c7, p.165-7), though he is referring to the "structural" means-ends relations in a work domain's "abstraction hierarchy", while we are concerned with "action" means-ends relations of tasks; in other words, relations between nouns vs. relations between verbs. While this distinction is interesting, it is somewhat tangential to our narrative and we feel that making mention of it in our paper might add an unnecessary dimension of complexity.//
>
>//Of more immediate concern to our paper were sections of Vicente's book pertaining to the analysis of tasks in terms of goals and constraints (inputs and outputs), to be distinguished from strategies for executing these tasks; we find that this characterization complements the why-what-how structure of our typology. The revised paper makes mention of this parallel in ᮠ//
*sequences of interdependent tasks with inputs and outputs are analgous to subroutines in the the ''decision ladder'' with the ''how'' filled in for each subroutine, having descriptive rather than formative power.
[[Atlas.ti|http://www.atlasti.com/]] and [[NVivo|http://www.qsrinternational.com/products_nvivo.aspx]] are Windows only.
Downloading and trying out [[HyperRESEARCH|http://www.researchware.com/products/hyperresearch]] for the Mac
*Tutorial 1: beginning a study
*Tutorial 2: working with codes - select / delete (instance / global) / duplicate / add description (global) / annotate (local)
*Tutorial 3: working with cases - delete / select / copy / paste / create new
*Tutorial 4: analyzing codes and reporting - generating reports with source material embedded, links to source material in context
In class activity:
>''Question 1'': What are you looking for in a marriage partner? (female, age 28, no children, never married, flight attendant)
>
>''Answer'': I want to marry a pilot. I want him to be about six feet tall with dark hair and blue eyes. You know, I just can't see myself with someone who I'm not physically attracted to and I love guys with blue eyes.
>
>I want him to want a big family and a woman who wants to be a stay at home mom. I love working, don't get me wrong, but I really want to marry someone who I can understand their career and support them in their career because I understand it.
>
>I'm a flight attendant and I've been working with pilots for about ten years now.
>
>I also want to stay at home and take care of my kids like my sister did. Her kids turned out great. So, I really believe in staying home and taking care of kids.
>
>I've actually dated two pilots lately and am dating one right now. So, I know I want to marry a pilot.
>
>I want someone who is really into family and doing family vacations and entertaining out parents and friends. You know, family oriented.
A [[Wordle|http://www.wordle.net/]]:
[img(90%, )[http://cs.ubc.ca/~brehmer/research/wordle.png]]
!!Holtzblatt and Jones '93:
!!!!Source: <<cite Holtzblatt1993 bibliography:Bibliography>>
Contextual Inquiry (CI) is a participatory design technique used in general-purpose design cycles subject to time and resource constraints. The technique is useful for gainin appropriate and helpful information about users' work practices. It treats //usability// not as attribute of a tool but as an attribute of interaction with a tool, whether that interaction helps or hinders existing practices. It focuses on the //work-of-the-work// rather than on the //work-of-the-tool//; the latter is rarely the focus of users during the course of their work.
>//people are engaged in doing work; they are not simultaneously reflecting upon their experiences of doing work.//
During a contextual inquiry, it is essential to speak with people during ongoing work, and not a reflective general summary of their work, which leads to discussion of abstractions and non-specific issues, which are in themselves not useful for driving subsequent design. Interviewees should be encouraged to articulate their work practices as they work. Conversations need to go both ways, a partnership.
>//the farther away we are from actual work, the more abstract the descriptions of work become.//
The user is the expert. Asking questions is acceptable, we are not the expert expected to think of technical solutions to users' problems. Users should be asked to explain new information and their actions. Interrupting users allows users to reflect on their experience at the time of their experience. Open-ended questions can help drive the conversation initially; it is natural for the focus to shift, attending to surprises and contradictions. A focus on technogy can be both revealing and concealing: a focus on broader work practices is encouraged. Design ideas emerging during the dialogue should be discussed rather than kept private to the designer, who may become distracted and unable to focus on what the user is saying.
>//only through dialogue can designers become aware of users' experiences of work and tool use 䠯scillation between engagement and reflection ࣡nnot get information if we do not interrupt 祶er, we do not talk all the time, we also watch in silence.//
CI is an adapatation of field methods from psychology, anthropology, sociology, and [[Grounded Theory]].
The authors provide logistical guidance for conducting CIs in small groups. Reviewing video and audio recordings together with users immediately afetr a CI can help when interruptions disrupt work practices. They also provide guidance for analyzing CI data in small groups by means of affinity diagramming. They conclude with ideas for appying CI at vaarious stages of the technology design cycle, integrating with other participatory design techinques.
!!Dix et al. '04:
!!!!Source: <<cite Dix2004>>: HCI text p. 471-472:
An in-depth 2-3 hour interview and observation of a user or user group in their regular work context, performing their regular tasks, adhering to the routines and materials/artifcats they use normally. Nonverbal as well as verbal communication is recorded. The researcher will ask questions to clarify what the user(s) is/are doing, why it is being done, and how the task fits into larger contexts. The user(s) use the [[Think Aloud Protocol]] to explain their actions as they are being carried out.
This method is in contrast to user interviews taking place in a location other than that of the user's workplace, or pure observation or video studies where the user doesn't describe their actions, nor does the researcher ask questions. In this way it differs from pure ethnographic research, as it doesn't take an open-ended view - its intention is to understand and interpret, to acknowledge and challenge the user(s) being studied - their aim is to design a new system, whereas ethnographic research is open-ended. It is more akin to a master/apprentice relationship than an observer/observed relationship.
Several models have been developed to capture what is important in the work context, focusing on sequence of actions (steps towards task completion), physical layout of the workplace and its influence on practices, flow (lines of coordination/communication), culture (work culture, policy, code of behaviour, expectations, values), and artifacts (structure, use of artifacts within work process).
Results must be transcribed, consolidated, and interpreted as soon as possible following an inquiry. Commonalities should be identified, [[Affinity Diagramming]] is useful here. Themes which are not predetermined arise out of the data, as in [[Grounded Evaluation]]. The result is a representation of constraints (physical and cultural), task sequences, artifacts, communication channels
!!Sedlmair et al. '12 (DSM):
!!!!Source: <<cite Sedlmair2012>> On CI vs. fly-on-the-wall (Think Aloud):
>//Most users do not accurately introspect about their own data analysis and visualization needs, making these needs invisible to themലied [the fly-on-the-wall] method in one of our projects 䠦ound it ineffective in a design study context as the complex cognitive tasks addressed by design studies are difficult to silently observe: in data analysis many things happen within the head of the user. While the methods of just talking and fly-on-the- wall provide some interesting information, expecting them to work alone is a pitfall. We have found good results with contextual inquiries [<<cite Holtzblatt1993>], where the researcher observes users working in their real-world context and interrupts to ask questions when clarification is needed.//
Similar to [[Principal Component Analysis]] in principle, this [[Dimensionality Reduction]] technique is particularly used for categorical data. Should numerical data occur in the dataset, each unique numerical value is treated as a distinct category (or a partition of values becomes categories). All data should be non-negative and on the same scale: rows and columns are treated equivalently. Correspondence Analysis (CA) works by decomposing a ''chi-square table'' between the variables and the objects (points)
''~DimStiller'' can conduct CA and display the results in a 2D image. However, when categories have close to identical profiles, category label overlap may occur in the ''symmetric map'' of the categories
!!!!Source:
*[[wikipedia article on CA|http://en.wikipedia.org/wiki/Correspondence_analysis]]
*[[DimStiller readme|http://www.cs.ubc.ca/~sfingram/dimstiller_readme.html]]
!![Links]
**[[Anyone can do it. Data journalism is the new punk|http://www.guardian.co.uk/news/datablog/2012/may/24/data-journalism-punk#_]]
From JS:
*[[In the age of big data, data journalism has profound importance for society|http://radar.oreilly.com/2012/03/rise-of-the-data-journalists.html]] by Alex Howard, Government 2.0 Washington Correspondent for O'Reilly Media
*[[Sarah Cohen|http://fds.duke.edu/db/Sanford/faculty/sarah.cohen]] on Orientation - NICAR talk on text analytics
*[[A Story-based Approach to Making Sense of Documents|http://www.actapress.com/Abstract.aspx?paperId=451928]], DOI: [[10.2316/P.2011.747-013|http://dx.doi.org/10.2316/P.2011.747-013]] by Eric Bier, William Janssen, Patricia Wall, Jonas Karlsson, Tong Sun, Wei Peng, and Zahra Langford. In Proceeding (746) Internet and Multimedia Systems and Applications / 747: ~Human-Computer Interaction - 2011
*[[The Data Journalism Handbook|http://shop.oreilly.com/product/0636920025603.do?sortby=publicationDate#]] by Jonathan Gray, Lucy Chambers, Liliana Bounegru (O'Reilly, 2012): [[free web version|http://www.datajournalismhandbook.org/1.0/en/]]
*[[google rating guide|http://www.seomoz.org/blog/16-insights-into-googles-rating-guidelines]]
*[[google search quality rater interview|http://searchengineland.com/interview-google-search-quality-rater-108702]]
*[[NICAR|http://www.ire.org/nicar/]] [[conference report|http://gov20.govfresh.com/the-expanding-world-of-open-data-journalism/]]: The culture, people, tools and preoccuptaions of the data journalism world.
*[[datadrivenjournalism.net|http://datadrivenjournalism.net/]]
*[[beer for data dave warner synergy strike force|http://blog.geoiq.com/2009/08/10/camp-roberts-exercise-and-the-afghanistan-elections-creating-a-geo-stack-for-humanitarian-relief/]] [[youtube|http://www.youtube.com/watch?v=kuayC90hvJk]] (one beer per folder)
*[[How Overview turns Documents into Pictures|http://overview.ap.org/blog/2012/06/how-overview-turns-documents-into-pictures/]]
Digital Humanities:
*[[Digital Journalism and Digital Humanities|http://www.dancohen.org/2012/02/08/digital-journalism-and-digital-humanities/]] by Dan Cohen
*[[BenSchmidt's digital humanities blog|http://sappingattention.blogspot.com/]]: [[Ben Schmidt|http://bmschmidt.wordpress.com/]]
*[[Orbis: The Stanford Geospatial Network Model of the Roman World|http://orbis.stanford.edu/#]]
*[[The Hermeneutics of Screwing Around; or What You Do with a Million Books|http://www.playingwithhistory.com/wp-content/uploads/2010/04/hermeneutics.pdf]]
From CWA:
*[[Truth, documents and data journalism鳴ory|http://www.reporterslab.org/cw-anderson/]], [[Reporters' Lab|http://www.reporterslab.org/]]
*[[The Things That Tell Us Whatⵥ (a Little Research Manifesto)|http://journalismschool.wordpress.com/2011/03/11/the-things-that-tell-us-what-is-true-a-little-research-manifesto/]]
*Additional reads:
**[[From Indymedia to Wikileaks: What a decade of hacking journalistic culture says about the future of news|http://www.niemanlab.org/2010/12/from-indymedia-to-wikileaks-what-a-decade-of-hacking-journalistic-culture-says-about-the-future-of-news/]] by CW Anderson, [[Nieman Journalism Lab|http://www.niemanlab.org/]]
**[[To build a digital future for news, developers must be able to hack at the core of old systems|http://www.niemanlab.org/2011/03/matt-waite-to-build-a-digital-future-for-news-developers-have-to-be-able-to-hack-at-the-core-of-the-old-ways/]] by M Waite, [[Nieman Journalism Lab|http://www.niemanlab.org/]]
**[[Audience Atomization Overcome: Why the Internet Weakens the Authority of the Press|http://archive.pressthink.org/2009/01/12/atomization.html]] by J. Rosen, [[PressThink|http://pressthink.org/]]
**[[The New Precision Journalism|http://www.unc.edu/~pmeyer/book/]] (book), [[Public Journalism and the Problem of Objectivity|http://www.unc.edu/~pmeyer/ire95pj.htm]] by P. Meyer
**NYT: [[A Selection From the Cache of Diplomatic Dispatches|http://www.nytimes.com/interactive/2010/11/28/world/20101128-cables-viewer.html?ref=wikileaks#report/cables-09KABUL3068]], [[All the Aggregation That鴠to Aggregate|http://www.nytimes.com/2011/03/13/magazine/mag-13lede-t.htm?_r=2]] by B. Keller, [[WikiLeaks|http://topics.nytimes.com/top/reference/timestopics/organizations/w/wikileaks/index.html?scp=1-spot&sq=wikileaks&st=cse]]
**Huffington Post: [[Why The New Republic is Wrong on Aggregation|http://www.huffingtonpost.com/robert-teitelman/the-new-republic-on-aggre_b_833105.html]] by R. Teitelman
From RR:
*[[psychology of intelligence analysis|https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/books-and-monographs/psychology-of-intelligence-analysis/index.html]]
*[[rensink cognitive systems course featuring heuer text|http://ling75.arts.ubc.ca/cogs/cogs303/]]
!![Howard2012] - Profiles of Data Journalists
[[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/data-journalists-data-markets-sports-analytics.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
>//Given the reality that those practicing data journalism remain a tiny percentage of the world's media, there's clearly still a need for its foremost practitioners to show why it matters, in terms of impact.//
[[Rise of the Data Journalists|http://radar.oreilly.com/2012/03/rise-of-the-data-journalists.html]]
*[[The Homicide Watch|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-9.html]] - NPR State Impact's Chris Amico and Homicide Watch's Laura Norton Amico [12.03.17]
*[[The Storyteller and The Teacher|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-8.html]] - Sarah Cohen (Knight prof. Journalism at Duke U) and USA Today's Anthony ~DeBarros [12.03.08]
*[[The Hacks Hacker|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-4.html]] - Matchstrike's Chrys Wu [12.03.08]
*[[The Data Editor|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-6.html]] - Virginan Pilot's Meghan Hoyer [12.03.06]
*[[The Daily Visualizer|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-5.html]] - NPR State Impact's Matt Stiles [12.03.06]
*[[The API Architect|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-4.html]] - NYT's Jacob Harris [12.03.05]
*[[The Visualizer|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-3.html]] - AP's Michelle Minkoff [12.03.02]
*[[The Human Algorithm|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-2.html]] - LA Times' Ben Welsh [12.03.02]
*[[The Elections Developer|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist-1.html]] - NYT's Derek Willis [12.03.01]
*[[The Long Form Developer|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] - [[ProPublica|http://www.propublica.org/]]'s Dan Nguyen [12.03.01]
Questions:
*//Where do you work now? What is a day in your life like?//
*//How did you get started in data journalism? Did you get any special degrees or certificates?//
*//Did you have any mentors? Who? What were the most important resources they shared with you?//
*//What does your personal data journalism "stack" look like? What tools could you not live without?//
*//What data journalism project are you the most proud of working on or creating?//
*//Where do you turn to keep your skills updated or learn new things?//
*//Why are data journalism and "news apps" important, in the context of the contemporary digital environment for information?//
!![Anderson2013] - Ethnography in Journalism
<<cite Anderson2013 bibliography:Bibliography-Overview>> uses an ethnographic methodology guided by [[Actor-Network Theory|http://en.wikipedia.org/wiki/Actor%E2%80%93network_theory]] to study collaboration and technology usage in two case studies, fields in the big-data flux: journalism and electoral politics. The theory accounts for activity at both micro and macro levels. Actors can be people and technology (human and nonhuman), and often artifacts generated at one level (such as electoral maps or newsroom content management systems) become actors at another, delegating further activity and/or recruitment of additional actors, development of further artifacts. Small actors / artifacts can have large ramifications at other levels.
!!!!Comments & Questions
*[[ANT|http://en.wikipedia.org/wiki/Actor%E2%80%93network_theory]] terminology:
**''actant'': //(in literary theory) a person, creature, or object playing any of a set of active roles in a narrative://
**''performativity'': //performativity creates social order//. from Wikipedia: //Performativity is an interdisciplinary term often used to name the capacity of speech and language in particular, as well as other non-verbal forms of expressive action, to perform a type of being. It is a forum, a performative act, a ritual, a social action that is omnipresent and without restriction; it extends socially, beyond constraints of system or structure. It is the construction of identity or position through active expression. Finally, it comprises the locution (the language - syntax, phonetics, reference, etc.), the illocutionary force (the social function of the locution - for example, if something is avert attention, the illocutionary force is the intention to avert attention), and the perlocutionary effect (the effect of the illocutionary force - in using the same example, if the person using language to avert attention was successful in distracting that receiver).//
**''black box'': //provisionally assembled actor-network that has been stabilized to such a degree that it appears solid and unproblematic//
**''immutable object'': and object that is //unchanging over time or unable to be changed//
**''centres of calculation'': //locales (not always tied to physical spare) from which knowledge and facts are both produced and administer, usually accordion got the logic of a particularly Machiavellian form of rationality and instrumentality//
**''immutable mobile'': // durable, difficult to transform objects that are easily transportable across space, thus exerting control and extending particular forms or rationality from the centre to the periphery, the primary product of centres of calculation//
**''enrolment'': //process by which actors strive to align with other actors with their interests, enrolling them into a network//
**''punctualization'': //refers to when a single actor or artifact gathers disparate networks together in a way that elides their complexity and thus affords easy action.// (i.e. maps and political work)
*addressing big issues relating to transparency within the tangled web of modern journalism, the roles of collaboration and technology, how this type of journalism is received by fellow journalists and the news-reading public, and where news production is heading
>very interesting work, especially with regards to data collection and analysis methods. I appreciate seeing individuals and processes characterized using a different methodological lens. Until recently, my research projects have been largely quantitative and empirical in the post-positivist tradition - controlled lab studies and surveys - but my current research questions have me looking to other fields for methodological insight. As of late, I've been doing a lot of reading about grounded theory, ethnography, and action research. I recently completed a graduate course in qualitative research methods, and I'm now trying to look at my research questions from an interpretive or critical viewpoint. The HCI field, still predominantly guided by the post-postivist thinking of cognitive psychology and human factors engineering, requires more rigorous instances of this type of work, so it's refreshing to see good examples of it carried out within research in other disciplines. Your methods paper using [[Actor-Network Theory|http://en.wikipedia.org/wiki/Actor%E2%80%93network_theory]] is particularly interesting for this reason.
>
>I'd also add that I enjoyed reading your description of political maps, how they are both actors and objects. This complements my earlier understanding of maps, which comes from a perceptual psychology and geospatial design standpoint.
!References
<<bibliography Bibliography-Overview showAll>>
!Tasks
*Are my dimensions meaningful? Can some be culled, filtered or combined?
*How do my dimensions relate to one another? Do the dimensions actually express a smaller subset?
>//"Does the data reside in a lower-dimensional manifold whose coordinate axes are best represented as a linear or nonlinear combination of the input dimensions?"// - ~DimStiller (<<cite Ingram2010 bibliography:Bibliography-DR>>)
*Are my clusters real? Clustering can inform the quality of dimensionality reduction, and vice versa.
Users are often mislead when it comes to choosing and applying Dimensionality Reduction (DR) techniques. There is likely a large middle-ground class of user who has domain knowledge and a detailed understanding of their high-dimensional data set, but lacks the math/stats background to understand DR techniques and how to apply them.
!//Attribute Reduction Methods// or //Column Reduction Techniques//
Reducing the number of attributes (or columns) can help uncover structure in the data: to verify or find meaningful clusters, or to verify/find the true meaningful dimensionality of the data. This can be done by eliminating (filtering out) attributes, or by aggregating them. Goals include finding a faithful representation of the data in a lower-dimensional space, capturing characteristics of the data that are important to the goals of the observer (finding/verifying clusters/dimensionality), while eliminating noise.
''Orthographic projections'' (different perspectives), ''slicing'' and ''cutting'' (i.e. medical imaging, cross sections) are ''Camera metaphors'' for reducing the number of dimensions. Attributes can also be ''filtered'' based on a derived attribute (a ''metric'', such as the attribute's variance). They can also be ''aggregated'' into fewer dimensions, wherein a single dimension could be representative for the group, or the dimension could represent the average of the grouped dimensions.
Reducing to ''synthetic dimensions'' is useful for uncovering ''latent variables'' (or ''intrinsic variables''), hidden structure, or for discovering the ''true dimensionality'' of the data set. These mappings from original to synthetic dimensions can be linear combinations of original dimensions, as in [[Principal Component Analysis]], or they can be [[nonlinear mappings|http://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction]], as in [[Multidimensional Scaling]] (MDS).
Methods to display data with reduced dimensionality as ''scatterplots'' produce layouts that are ''affine invariant'', meaning that they can be rotated, scaled, or mirrored while maintaining the same meaning. The layouts can provide insight into large-scale cluster structure. Small-scale structure is often not reliable. Other visualization methods include ''scatterplot matrices'' (~SPLOMs), and ''landscapes'', however the latter suffer from 3D perspective issues.
!!Dimensionality Reduction Techniques
[>img(33%, )[A Scree Plot|http://janda.org/workshop/factor%20analysis/SPSS%20run/SPSS08.gif]]
*''PCA'': [[Principal Component Analysis]] - finding dimensions which are linear combinations of original dimensions that capture the most variability in the data, wherein the first dimension captures the most variability, while the second and subsequent dimensions (each being orthogonal to the preceding dimensions), capture the most of the remaining variability in a diminishing fashion. A [[Scree Plot]], a histogram of dimension variability, will indicate the number of dimensions contributing the most variability. An observer can decide a threshold which accounts for most of the variability, thereby discarding remaining dimensions. This can reduce noise in the data.
*''SVD'': [[Singular Value Decomposition]] - PCA is equivalent to SVD once the mean of each variable has been removed, which may or may not be desirable, particularly when the data is sparse.
*''FA'': [[Factor Analysis]] - original dimensions are expressed as linear combinations of a small number of hidden or latent dimensions. The motivation often stems from the presence of clusters of dimensions that are in amongst themselves highly interrelated, but not related to other clusters.
*''LLE'': [[Local Linear Embedding]] - a nonlinear approach that analyzes overlapping local neighborhoods in order to determine local structure.
[>img(33%, )[A Swiss Roll|http://c13s.files.wordpress.com/2010/04/swiss-roll.png?w=700]]
*''MDS'': [[Multidimensional Scaling]] - beginning with a dissimilarity matrix, the dimensionality reduction is a projection of the data to a lower-dimensional space, preserving pairwise distances as well as possible, measured by some objective function. In ''Metric MDS'', dissimilarities are continuous, while in ''Non-metric MDS'', dissimilarities are categorical or ordinal. Classical MDS for Euclidean distance is equivalent to PCA. Different results can be obtained each time it runs.
**''~FastMap'' - fast MDS (linear complexity), operating incrementally: pairwise distances are computed using cosine similarities. Objects are projected onto a (//n- 1//) dimensional subspace
**''ISOMAP'' - when points have a complicated, non-linear relationship to one another, MDS and PCS break down. ISOMAP can handle such data. A 2D data set in a 3D space (i.e. the [[Swiss Roll]] surface), nearest-neighbours are computed using the geodesic distance (along the surface), rather than using Euclidean distance - it can flexibly learn a broad class of nonlinear manifolds. Only the geodesic distances reflect the true low-dimensional geometry of the manifold, whereas classical MDS and PCA just see the Euclidean structure. ISOMAP preserves the intrinsic geometry.
*''ICA'': [[Independent Component Analysis]] - another factor rotation method
*''PP'': [[Projection Pursuit]] - finds most interesting projections in multidimensional data, useful for non-Gaussian datasets
*[[Manifold Learning]] - the process by which high dimensional data (HDD) is described in a low-dimensional basis (i.e. ''ISOMAP'').
*''Regression''
*[[t-SNE]] - t-Distributed Stochastic Neighbour Embedding, and multiple maps ~t-SNE - used for visualizing non-metric similarities, such as word similarities or co-authorship graphs where triangle equalities do not hold to be true, captures centrality of objects in 2D space
!!!Sources:
<<bibliography Bibliography-DR showAll>>
Thompson, D. R. //Manifold learning with applications to object recognition//. Presentation for Advanced Perception.
[[Nonlinear Dimensionality Reduction|http://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction]] (wikipedia)
!A research proposal for [[EPSE 595]]
>Write a brief proposal (8 - 10 pages, double-spaced) for an interpretive or critical research project on a topic of interest to you. The proposal should include the research question(s), description of the context you propose to conduct the research in, methodology, data collection methods and sources, and at least three references that you will build on in doing the research. The proposal should include more than one data collection method.
This project will also form the basis of part of my RPE, a 4 month research stint.
!!Introduction
In the fall of 2010, the [[WikiLeaks organization|http://en.wikipedia.org/wiki/WikiLeaks]] released [[nearly 400,000 text documents|http://goo.gl/TAgOs]] relating to the conduct of US armed forces and independent security contractors during the war in Iraq. Since that time, specialized investigative ᠪournalists�reported on what was contained in this vast deposit of documents, which included startling information regarding civilian casualties, friendly fire incidents, and observed breaches of protocol. My proposed research, in brief, asks how data journalists 彨 deposits, how they seek information to support or refute prior findings, or how they come to discover unanticipated yet newsworthy stories hiding in the data.
The motivation for this research proposal reflects the growing trend of large collections of emails, reports, and other documents being ।�d�ased, or declassified by corporations, government agencies, and other organizations, such as ~WikiLeaks. [[Fellow journalists and the news-reading public deserve transparency|http://www.reporterslab.org/cw-anderson/]] when it comes to the methods of data journalists who investigate these collections. Additionally, developers of data analysis applications require a better understanding of journalists as potential users.
Aside from being a consumer of news media, I have no background in journalism. Rather I represent the interests of application developers, and I am sensitive to theoretical concepts relating to the design of tools for supporting data analysis. While this theoretical repertoire is useful for understanding low-level perceptual activity contributing to how individuals interact with data displays [<<cite Amar2005 bibliography:Bibliography>>, <<cite Shneiderman1996>>], or for understanding high-level domain-agnostic abstractions relating to information foraging and sense-making [<<cite Amar2004a>>, <<cite Pirolli2009>>], I am faced with a gap in the middle. How can I characterize the data analysis process within the context of a domain such as journalism? Moreover, how does this process play out with a specific type of data, in this case being collections of text documents? Thus I seek a middle-level theory to explain this process.
!!Research questions
My predominant research question asks: What is document mining? That is, how do journalists conduct data analysis when faced with more documents than they could possibly read in a year, let alone in time to meet a deadline?
Several additional questions follow from this: namely, what constraints do these journalists face in the process of their work? What tools do they use and how do they use them? Do they collaborate with other people, and if so, how do they collaborate?
Finally, this work will determine how document mining compares to other analytical processes in data journalism, when the data is comprised of numbers rather than text documents, which could include large financial databases or historical measurements. I will also make comparisons between document mining and other processes of investigative journalism, as well as with processes of data analysis characterized in other domains, such as business intelligence and law enforcement.
!!Research context
Journalists engaging in document mining are found in newsrooms around the world. Unfortunately, like many busy professionals, they are often working under tight deadlines, and have little time to participate in academic research. However, I am lucky enough to know someone 䨥 insideꓠis a computer-scientist-turned-journalist now employed by the Associated Press (AP), based out of New York City. Working in collaboration with my research group, he has developed [[Overview|http://overview.ap.org/]], a robust data visualization application for document mining, recently made available as a free download on the AP墳ite. He is currently pitching Overview to journalists via conferences, workshops, and social media. Buzz surrounding Overview is starting to grow in the data journalism community.
JS is our gatekeeper to research participants, as his potentially useful application provides an incentive for journalists participate in our research. As a result, we have an opportunity to satisfy two research goals: (1) assess whether Overview is usable and useful, as well as how it fits within existing document mining workflows; and (2) characterize the process and context of document mining, with and without this new application. While this proposal focuses on the latter goal, data collection corresponding to both goals will occur simultaneously. Furthermore, it is my intent that in working toward the second goal, my findings will contribute to the future development of Overview and other applications like it.
Due to the distributed nature of this research, logistical constraints will keep me from visiting individual journalists in newsrooms. Thus my data collection will occur at a distance, over the phone and online.
!!Methodology
A need to characterize the process of document mining among journalists necessitates a grounded theory approach [<<cite Charmaz2006>>]. This approach is in turn informed by an interpretivist, symbolic interactionist theoretical perspective and a constructionist epistemology [<<cite Crotty1998>>]. That is, I intend to focus on the language used by journalists to describe this process, and construct a shared interpretation of this process based on interactions with research participants and the data they generate.
The constant comparative method of grounded theory will allow me to flexibly make comparisons between the process of document mining with other journalistic processes, as wella as processes relating to data analysis in other domains. Comparisons will also be made between journalists, between newsrooms, and over periods of time. For instance, I will be comparing the process of document mining both before and after the introduction of Overview, the new visualization tool.
A further justification for the use of a grounded theory methodology is that my initial research questions are not theoretically deduced hypotheses. Rather, my questions are informed by sensitizing concepts and assumptions held within my domain. It is these sensitizing concepts that allowed me to frame the data collection methods, particularly a preliminary set of interview questions.
These sensitizing concepts include the notion that data analysis, document mining being an instance of which, occurs in stages. Data analysis may involve stages of hypothesis generation, each necessitating an exploration of the data without a particular set of questions in mind, save perhaps 䯩ng on here?䨥r times there may be stages of hypothesis validation, where the goal is to support or refute prior evidence. These stages necessitate a directed search within a subset of the data, or a comparison between individual items or documents. Individuals may or may not engage in both types of stages during the course of a single investigation.
The products of data analysis are also among my sensitizing concepts. These products include 嫡!�ts of insight, serendipitous discoveries, and both optimal and suboptimal solutions to closed- and open-ended problems. Admittedly, these products of analysis are ill-defined constructs, and it will be necessary to attain our research participants䥲pretations of their meaning, as well as their native terminology.
Finally, these sensitizing concepts include the disentanglement of an individual谥rtise. By this I mean that an analyst may have expertise within a domain, expertise using specific analytical tools or techniques, and/or expertise regarding the data, its semantics and its provenance.
In my field of research, there exist several precedents for the use of grounded theory, or at least the use of methods inspired by a grounded theory methodology. The methodology has informed prior work which has characterized the data analysis processes of professionals in other domains, including architecture [<<cite Tory2008>>] and national security and intelligence [<<cite Kang2011>>]. There also exists a 室ed evaluation�ique for determining the effectiveness of visualization software when deployed in target contexts of use [<<cite Isenberg2008>>]. Both uses of grounded theory methods serve as inspiration for my proposed research.
!!!Data collection methods and sources
''Primary'': My primary data collection method will be intensive, open-ended interviews with journalists. These interviews will be teleconferences or group Skype chats. Both JS (at the AP in New York), and myself (at UBC) will have questions to ask interviewees, with his questions pertaining to the usability and utility of Overview. Audio from these interviews will be recorded for later transcription.
Following the methodology of grounded theory, I will not specify the number of interviews that I plan to conduct a priori. The final number of interviews will depend on how much theoretical sampling is required before achieving data saturation, the point where no new categories emerge; I will return to this point in the following section. The number of interviews will also depend on how many journalists download and use Overview, and among those, how many express a willingness to participate in interviews. Ideally, I would like to perform multiple interviews with each journalist in order to make comparisons over time, as their processes vary or change over time, before, during, and after using the new tool. However, this is an unrealistic plan. As mentioned above, these journalists will often be conducting their investigation and writing their story under a tight deadline, and will likely only have time to commit to an intensive interview after the story is written. As a result, I will rely on secondary data collection methods, such as follow-up email exchanges, to fill in some of the gaps. These methods are discussed in greater detail below.
Regarding the content of these interviews, I plan to keep the number of initial questions small. I have [[prepared a list of interview foci with a small set of representative questions|Overview Deployment Field Survey]] for each, composed according to guidelines for open-ended interviews [<<cite Fontana1994>>] and for interviews conducted in the context of a grounded theory study [<<cite Charmaz2006>>]. These foci correspond with the research questions mentioned above. In particular, I will attempt to ground the interview in the interviewee衭ple of document mining, one drawn from their prior experience. This will invite comparisons with other journalistic processes, as well as comparisons between their processes before and after their initial use of Overview.
Many questions are redundant and cross-referential, a deliberate choice, as I have no intention to ask all or even most of them in a single interview. An answer to one open-ended question is expected to answer many of the others; these foci and questions are more so a checklist than a script. I also expect this list of foci and questions to change as I conduct interviews, as a result of theoretical sampling and the possibility that early interviews will illuminate unanticipated themes and concepts.
''Secondary'': I plan to complement the interviews by eliciting texts and other information from journalists that I interview. Follow-up questions will be asked via email. I will also request copies of the notes journalists take during the course of their investigation. I expect that in many cases, journalists will be taking notes regardless of whether or not I ask to see them. I will also request information regarding the data, such as how many documents are contained in the document collection being investigated, how they tend to vary from one another, how many were read or skimmed during the course of their investigation, how many were discarded or ignored, as well as why individual documents were read, skimmed, ignored, or discarded. In cases where these documents are publicly available, I will examine the documents as well. Screenshots or pictures of annotated documents, journalist notes, and other analytical artifacts, such as spreadsheets and data visualizations, will also be requested.
Realizing the value of found data [<<cite Silverman2007>>], I will also collect several extant texts. In particular, I will collect the stories journalists write as a result of their investigations. I will examine the extent to which the document mining process is transparent in their articles, allowing for a comparison with their notes and the remarks they make during interviews.Finally, in cases where these stories are published online, I will also collect the reader perspective, via comments and discussion boards.
>@@color:#444bbb; ''SM'': //"All sounds good. is there any sort of tracking embedded in Overview? If so, you might also be able to see how journalists move within and between documents when data mining. This may be beyond what you want to do, but studying the pathways through data and documents might reveal something about decision-making, choices and conclusions (as evidenced in what the journalist ends up writing)."// @@
!!!Data analysis methods
Data collection and analysis for this project will occur concurrently. Interviews and artifacts collected from journalists will be subject to multiple iterations of coding, each calling upon the constant comparative method, the basis of grounded theory [<<cite Charmaz2006>>]. First, open coding will label the data, at the line or paragraph level, using words or short phrases used in the data. Next, tentative categories of codes will be generated, each with an explanatory rationale based on comparisons between code instances, recorded in memos. This will inform subsequent data collection and focused coding. The process of axial coding follows, a recoding of the data using the categories constructed. At this point, the process of theoretical sampling will direct me to specific data collection, using different interview foci or artifact collection criteria. As categories become refined and theoretical concepts emerge through the process of selective coding and memoing, I will begin to seek theoretical saturation, the point where no unexamined concepts are apparent. At this time, I will begin to construct a mid-level theory of document mining based on the relationships between concepts. This stage will also involve comparisons between my theory and other theories of data analysis, as it occurs in other domains and as it is described at higher levels of abstraction [<<cite Amar2004a>>, <<cite Pirolli2009>>].
Triangulation between researchers is highly effective during interpretive analysis [<<cite Mathison1988>>]. I will share my findings with JS. While he and I have different foci and research goals, we will both be engaged in participant interviews, and thus can compare notes. Additionally, his own journalistic expertise can also be called upon throughout the stages of my analysis.
I will also triangulate in terms of methods, in that I will take an alternative approach to thematic coding [<<cite Ryan2003>>]. This will involve an examination of word frequency, word co-occurrence, key words as used in context, linguistic connectors, and metaphors used. Extant texts and artifacts collected will also be analyzed in terms of their descriptive properties, as well as their intellectual and cultural values [<<cite Prown1982>>]. I will then compare the codes produced by these alternative techniques to the codes and categories generated via the grounded theory methods.
!!Outcomes and follow-on work
I anticipate two audiences to which I will report my findings. The first are readers of peer-reviewed academic publications and/or conference proceedings in the fields of human-computer interaction, information visualization, and visual analytics. The second audience for my findings are journalists and the news-reading public, so I plan to report my findings online, either via my own website or in collaboration with the AP.
I hope to apply my findings in the future development of Overview and other applications like it. Finally, I anticipate that an examination of what makes Overview ultimately successful or unsuccessful will call for a critical inquiry of existing values and standards in journalism, as well as existing theories of data analysis.
>@@color:#444bbb; ''SM'': //"what a pleasure to read. Clear, well reasoned and absolutely doable. Good luck with the research."//@@
!!References
<<bibliography>>
!Journal & Course Schedule:
Here is a journal of topics relating to EPSE 595, an introductory course to qualitative research taught by [[Sandra Mathison|http://blogs.ubc.ca/qualresearch/2011/09/28/research-design/]]:
|!Date |!Topic |!Methodology |!Readings |!Workbook |
|''Jan 4'' |>|Introduction to interpretive and critical research | | |
| ''Foundations of Research'' |>|>|>|>|
|''Jan 11'' |>|[[Epistemologies|Elements of Research: Epistemology]]: [[Objectivism|Epistemology: Objectivism]], [[Constructionism|Epistemology: Constructionism]] | | |
|''Jan 18'' |>|[[Social Constructivism: Interpretivism|Epistemology: Constructionism]] |<<cite Crotty1998 bibliography:Bibliography-EPSE595>> ch. 1-5 | |
|''Jan 25'' |>|[[Social Constructivism: Critical Inquiry|Epistemology: Constructionism]] |<<cite Crotty1998>> ch. 6-7 | |
|''Feb 1'' |>|[[Reading Interpretive/Critical Research]] |<<cite Pascoe2007>> | |
|''Feb 8'' |>|[[Research purpose, design, questions|Qualitative Research: Purpose, Design, & Questions]]; [[Research Ethics]] |<<cite Silverman2007>>, <<cite Tri-Council2010>> ch. 9-10, <<cite Freeman2009>> ch. 2, 3, 5 | |
| ''[[Research Methodologies & Methods]]'' |>|>|>|>|
|''Feb 15'' |[[Participant Observation]] |[[Grounded Theory|http://prezi.com/johwoujelpfk/grounded-theory/]]: [[notes|Grounded Theory]] |<<cite Becker1957>> | [[#1|EPSE 595 Workbook 1]] |
|''Feb 22'' | //Reading Week// |>|>|>|
|''Feb 29'' |[[Interviewing]] |[[Ethnography|http://prezi.com/ftcr-ez_736n/ethnographies/]]: [[notes|Ethnography]] |<<cite Fontana1994>> | [[#2|EPSE 595 Workbook 2]] |
|''Mar 7'' |[[Group interviews|Interviewing]] |[[PhotoVoice|http://prezi.com/_m_lndsuctib/photovoice/]] |<<cite Krueger2002>> | |
|''Mar 14'' |[[Material & Environmental Data]] |[[Narrative Analysis|http://prezi.com/i9c0uupsydvi/narrative-analysis/]] |<<cite Mathison2009>>, <<cite Prown1982>> | [[#3|EPSE 595 Workbook 3]] |
| ''Data Analysis & Representation'' |>|>|>|>|
|''Mar 21'' |[[Organizing and Making Sense of Data]] |[[Action Research|http://prezi.com/0lc7aiia6rlx/action-research/]] |<<cite Mathison1988>>, <<cite Ryan2003>> | [[#4|EPSE 595 Workbook 4]] |
|''Mar 28'' |[[Computer Assisted Data Analysis, Data Displays]] |[[Phenomenolgy|http://prezi.com/gudzqww9jcv8/phenomenology/]] |//(Download and try out any of the demo analysis software versions)// - [[HyperRESEARCH|http://www.researchware.com/products/hyperresearch]] | |
|''Apr 4'' |>|[[Representing Knowledge]] |<<cite Baff1997>>, <<cite Sandelowski1998>>, <<cite KadriZald2004>> | [[#5|EPSE 595 Workbook 5]] |
|''Apr 11'' | //Research proposal//: [[Document mining in data journalism]], [[(Interview Foci, Questions)|Overview Deployment Field Survey]] |>|>|>|
<html>
<object id="prezi_60f3a8de49793b42c656fefca1c57d772bebc870" name="prezi_60f3a8de49793b42c656fefca1c57d772bebc870" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"><param name="movie" value="http://prezi.com/bin/preziloader.swf"/><param name="allowfullscreen" value="true"/><param name="allowFullScreenInteractive" value="true"/><param name="allowscriptaccess" value="always"/><param name="wmode" value="direct"/><param name="bgcolor" value="#ffffff"/><param name="flashvars" value="prezi_id=60f3a8de49793b42c656fefca1c57d772bebc870&lock_to_path=0&color=ffffff&autoplay=no&autohide_ctrls=0"/><embed id="preziEmbed_60f3a8de49793b42c656fefca1c57d772bebc870" name="preziEmbed_60f3a8de49793b42c656fefca1c57d772bebc870" src="http://prezi.com/bin/preziloader.swf" type="application/x-shockwave-flash" allowfullscreen="true" allowFullScreenInteractive="true" allowscriptaccess="always" width="550" height="400" bgcolor="#ffffff" flashvars="prezi_id=60f3a8de49793b42c656fefca1c57d772bebc870&lock_to_path=0&color=ffffff&autoplay=no&autohide_ctrls=0"></embed></object>
</html>
!References:
<<bibliography Bibliography-EPSE595 showAll>>
~BibTeX: [[Bibliography-EPSE595]]
A. How would you describe your epistemology in terms of the Crotty鳣ussion of objectivism and social constructionism? Explain why. Be sure to give concrete examples (for example: beliefs, experiences, things you have read, the way you think about things) to illustrate your epistemology. Spend your time exploring your own position rather than explicating these epistemologies. (Remember, I⥡d the book too!) It is helpful to think about how each epistemological perspective characterizes the nature of knowledge, truth, what is known when we know, the relationship between the knower and the known, the limits of knowledge. (2-3 pages)㭔OC project - interruptions
**domestic interruption field study
**what constitutes an interruption, a distraction
**ever-changing contexts: the home is not a stable context
*basic research - graphical perception, mid-level processing, attention and memory (objective)
*usability testing, heuristic evaluation (objective)
*requirements analysis (towards a shared understanding of use cases, social constructivist)
**task analysis, a shared understanding of a task
*utility assessment / adoption and post-deployment studies (a shared understanding of what constitutes success and failure, useful and useless, social constructivist)
*other issues in data analysis, particularly relevant in ~InfoVis (and to a lesser extent, all of HCI, social constructivist - see conceptual map in (B))
**Insight (a mutual understanding with a domain expert)
B. Identify a research topic you care about. Develop a visual conceptual map that illustrates what you think the components and relationships within this topic are. You can do this based on what you already know, but it may be helpful to skim some literature if you are not familiar enough with the research on this topic. (Google Scholar can be your friend here.)ght / discovery / serendipity
**insight-based methodologies (Saraiya and North)
**as means or as ends
*learning (about a dataset, a tool, a task)
**domain expertise
**task expertise
**exploratory data analysis vs. direct queries
*problem solving
**hermeneutical approaches (Klahr & Simon)
**laboratory studies
**observational studies
**computer modeling
*creativity
**constraint-based creativity (Jennings) / optimization problems (well-defined problems)
**exploratory creativity (ill-defined problems, open-ended problems)
*analytical gaps (rationale, worldview) (Amar & Stasko)
C. Based on what you have done in (B) write three possible interpretivist/critical research questions. For one of those questions, elaborate on the focus of the research by developing sub-questions (which should be informed by your conceptual map).鳵alization should facilitate insight. But what constitutes insight, or an unit of discovery? Is insight the means or the ends? Can we establish a shared interpretation of insight that traverses domains? (Interpretivist)
*Why do existing high-dimensional visual analysis tools break down for abstract problem-solving tasks? (Critical)
**How can we better support these tasks?
**Are there analogous problems from other domains involving high-dimensional data analysis that can be applied here?
**Are there analogous problems from historical accounts?
*How does one develop expertise as a visual analyst? (Interpretive)
**how does one initially learn or explore a dataset?
**how does one learn abstract analysis tasks (that span multiple tools / techniques)
**how does one learn how to effectively use individual tools and techniques?
!!Part A
I毵nd that ~Human-Computer Interaction, a multidisciplinary field of study, inherits the dominant epistemologies of the disciplines that comprise it. I admit this is what drew me to the field, the opportunity to wear many hats: the cognitive scientist, the industrial/graphic designer, the computer scientist, and the ethnographer. Each role asks a different set of questions dependent on the current research goal and the stage of a project. As a result, meaning can be discovered at one stage, while it must be constructed at another.
!!!The cognitive scientist
As a cognitive scientist, I am interested in issues relating to perception, cognition, and behaviour. An understanding of these issues is critical for justifying the design of future technologies, or for explaining behaviour with existing technologies [2]. I could ask questions about low-level sensory perception, such as //n individual is presented with a simple [visual, auditory, or tactile] stimulus, what is their response?폲 I may ask about mid-level processing, about the recognition of patterns, of aggregates of simple stimuli (i.e. //㠴he set of points in a scatter plot suggest a correlation between variables X and Y?�nswering questions at these levels often have direct implications for a technology㡢ility: how efficiently and correctly can it be used? Finally, I may ask questions about higher-level processes, those spanning minutes or longer, about divided attention and an individual鮩ability to perform multiple tasks concurrently (i.e. //य software developers effectively write code while maintaining instant messaging conversations?�⥥n fortunate to participate in several research projects that ask these types of questions. In each case, the typical means by which these questions are answered is via post-positivist hypothesis testing and controlled laboratory experiments. In this sense human-computer interaction (HCI) researchers inherit the objective epistemology of cognitive psychology.
Several of my recent research projects in this highly post-positivist track have pertained to issues of multi-tasking. During the course of these projects I have begun to question the appropriateness of an objective post-positivist stance. A significant component of my M.Sc research [1] was devoted to studying user performance on early prototypes of a self-administered computerised web-based cognitive testing application, to be taken at home by older adults as an initial screening test for age-related dementias. Our research group acknowledged that the home is unlike a quiet clinic office, the setting where cognitive testing is presently conducted. In particular we assumed the home setting to be one in which distractions and interruptions could inhibit performance on such a cognitive test, a test that demands full attention and involves time-sensitive responses. Foreseeing these problems, we decided that a deeper understanding of the interaction between user age and interruption type. We believed this to be necessary for the purposes of designing interventions for preventing, detecting, and mitigating interruptions within the application itself. We selected an approach grounded in prior research in psychology and HCI: we conducted a controlled laboratory experiment in which research participants from different age groups performed cognitive testing tasks, interleaved with short interrupting working-memory tasks, intended to simulate a range of possible interruptions which might occur in the home. Our results were unexpected and largely inconclusive. Upon reflecting on this outcome I⥡lised how a strictly post-positivist perspective limited what we could study and how we could study it. For instance, we could not guarantee a shared understanding of what cognitive testing meant to our different groups of participants (I expect this to be a more sensitive and significant topic for our oldest group). Moreover, we did not have a shared understanding of what constituted an interruption, how it may occur in the home, and how one may respond to it naturally (as opposed to in an experimental setting), particularly when one燐itive health (a sensitive matter, of course) was being assessed.
I now acknowledge how higher-level cognitive processes, such as multi-tasking and handling interruptions, are highly dependent on context, on a mutual understanding of that context and the processes involved, that these must be constructed with our research participants. While I maintain that issues of low-level sensory perception and mid-level pattern recognition ought to be studied with a post-positivist perspective, the study of higher-level cognitive processes may call upon a mixture of theoretical perspectives.
!!!The designer
An understanding of what is aesthetically pleasing is an important asset for the HCI researcher or practitioner. I argue that this too, requires a mixture of epistemological perspectives. As we design interface technologies intended for human interaction, we rely on a history of established graphic and industrial design guidelines, as well as familiar interaction techniques, those that are known to provoke a positive response from users. Many of these positive responses are supported by experimental findings in the cognitive sciences. In a sense we know what is considered pleasing, elegant, intuitive, or natural because it correlates with what is efficient and accurate. We objectively know which guidelines and techniques work based on what sells, on market research and survey studies.
However when working in a new medium it becomes necessary to challenge the aforementioned constructs: of what is pleasing, intuitive, etc. Handheld touch screens and large shared public displays are recent examples of how new technological mediums have redefined what was previously considered to be an intuitive interaction, just as hypertext, the mouse, and the graphical interface did in decades prior.
Once again, the HCI practitioner, in their role as a designer, must be able to shift their epistemological perspective based on the medium they are currently working in. This could entail a constructionist perspective, observing individuals and groups as they engage with a new medium, as well as discussing and negotiating what these constructs have come to mean in light of their experiences.
!!!The ethnographer
A pivotal role of the HCI researcher-practitioner, one in which I岲ently assuming in my ~PhD studies, is one of an ethnographer. We exhaustively study users and their day- to-day context before designing a technological application. Similarly, we also study these users and their contexts after these applications have been deployed.
//Requirements analysis// is a process that begins when a client, often a representative for a group of employees or customers (the target users), approaches you with an ill- defined problem, a perceived need for a technological intervention. For instance, my research group is currently engaged with a journalist who represents information analysts who currently sift through large corpuses of text documents (i.e. government freedom of information requests, ~WikiLeaks㵭ent collections, etc.), in search of newsworthy items or trends. It is our task to study the existing workflow of these individuals and establish a common understanding of these tasks and their context. We then negotiate the requirements of a technological intervention, which problems or needs it will address, and the relative importance of these needs. A mutual understanding must also be reached in terms of whether such an intervention will replace an existing workflow, add to workflow, or if instead it will provide an alternative to the workflow.
At the other end of the project timeline, following the deployment of a technological application, we study whether and how users⫦lows have changed. We ask whether the intervention was successful, as well as what constitutes success and failure for each party involved. Similarly we examine whether the application was perceived to be useful, at the same time determining what constitutes the constructs usefulness and uselessness (in other words, utility).
Before development and after deployment, it is apparent, at least to me, that the HCI researcher-practitioner must maintain a constructionist perspective. While I expect that an objective survey approach could accumulate a large amount of information regarding workflows, use cases, application requirements, and perceived utility, too much is lost. There are simply too many constructs that need to be negotiated.
!!!Conclusion.
At a glance, it appears messy and if as though HCI researcher-practitioners, myself included, suffer from split-personality disorder. We adopt an objective perspective for one project and a constructionist perspective for another. Often this perspective changes within a project. This is a result of the multidisciplinary nature of the field. Iﴠworried about this. We may construct a use case that requires developing a novel interaction technique, one that must be evaluated objectively. If anything, I view this ability to shift perspectives as adaptive behaviour.
!!Part B.
As mentioned in Part A, I岲ently interested in how information analysts sift through and make sense of large amounts of data. A specific focus is on journalists and large collections of text documents (FOIA requests, ~WikiLeaks, etc.). More broadly, this interest involves comparing analyst behaviour and processes across domains (i.e. a journalist compared to a biomedical researcher). There are now countless industries and research areas working with large datasets, be it text or numbers.
My research group is one of many that develop applications that facilitate these processes, often in the form of interactive information visualisation tools. It is said that the goal of these tools is ''insight'' into the data. In the following visual conceptual map, I䥣ided to unpack this term and its related concepts (for sources and my inspiration, see References: Part B).
[img(90%, )[http://cs.ubc.ca/~brehmer/research/595w1.png]]
!!Part C.
#What constitutes insight, or a unit of discovery? When is insight the means (hypothesis generation) or the ends (hypothesis validation)? Can we establish a shared interpretation of insight that traverses domains? (Interpretivist)
#How do existing data analysis and visualization practices facilitate insight via different types of problem-solving strategies? Where do they break down? How is collaborative analysis and problem-solving supported? (Critical)
#How does one develop expertise as an information analyst? (Interpretivist)
##How does a novice analyst initially explore a dataset? How is this different from how an expert analyst would explore a dataset?
##How does collaboration between novice and expert analysts, or between novice analysts, effect the development of expertise?
##Are there commonalities across domains with regards to how one learns to use information analysis tools and techniques?
##How does one learn domain-independent analysis tasks (those that span multiple tools and techniques)?
!!Participant observation notes (written immediately post-hoc)
//Date//: Friday, February 24th, 2012, Time: ~8pm - ~8:30pm
//Place//: Zulu Record Store and Community Center, co-located with Videomatica DVD & Video Sales, West 4th Avenue Vancouver at Maple Street
//Familiarity//: I live across the street from this record shop. I have been into the store several times a year since moving to Vancouver in 2009. Zulu is an independent record store, offering both new and used ~CDs and ~LPs for sale, stereo equipment sales, and concert ticket sales.
They have recently merged with Videomatica, a video rental and sales store that was once located a block east. Videomatica no longer operates a rental service; they currently offer films on DVD for sale, specializing in independent, foreign, cult classic, and other niche genres of film. Prior to merging with Zulu, the only ~DVDs on sale at Zulu were of live concert films.
//Participant role//: on this occasion, I went to Zulu with my partner to browse for ~CDs to listen to in our car, specifically for music to listen to while driving to Whistler on weekend ski trips. We were looking for ~CDs in particular since we do not have good radio reception on the sea-to-sky highway, nor do we have the means to use an mp3 player with the car䥲eo system. My partner was hoping to find some upbeat music, as she has remarked that many of my ~CDs are too mellow, and not ideal for staying awake on the road after a day of skiing.
Like most times I frequent Zulu, I also browse the LP section, where most of tonight쩥ntele were.
!!!Description of Context:
(see sketch on following page)
*ground floor store in West 4th shopping district
*two street-facing doors, windows in between, windows partially occluded by posters for upcoming concerts, recent releases, ads, promotions, testimonials, a list of the staffᶯurite records released in 2011, used vinyl ~LPs along a display at the bottom of the windows
*inside, two large rooms, a dividing support wall running perpendicular to the street along the middle of the store
*in the west room, the LP section, waist-height shelves of 45rpm records at the front of the store, a shelf along the west wall, a shelf in the the middle of the room, the LP checkout *counter along the central wall at the front of the store, behind this along central wall, a shelf of records extending to an entrance in the central wall leading to the east side of the store
*rear of store峴 room, behind the entrance in the central wall to east room: the Videomatica section, a short vertical shelf parallel to the street, one behind this shelf along the west wall, another along the central wall, a large flat-panel TV screen above the checkout counter at the rear
*east room of the store, mostly ~CDs (new and used), magazines, a few ~LPs on a display rack, discount used ~CDs at the rear of the store, a sunken area at the front of the store, the CD checkout counter, a new releases display shelf by the east door
*an entrance and short set of stairs joins the two checkout counters, for staff only
*It is not easy to see the LP section from the CD section, and vice-versa, due to the central wall
[img(75%, )[http://cs.ubc.ca/~brehmer/research/595w2_1.png]]
!!!People:
M = male; F = female;
*A. Staff:
**Zulu clerk: ZF, caucasian, mid-late 20s, long brown hair
**Zulu clerk: ZM, mid-late 20s, caucasian, short, thin; short blond hair, no facial hair
**Videomatica clerk: male, late 20s - early 30s, caucasian, long, shoulder length brown hair, no facial hair; initially faced away from me when I entered the store, I had mistaken him for a woman.
*B. Clientele:
**M1 - Male, asian, pre-teen, left the store with M2 - his father?
**M2, asian, middle-aged, wearing coat, ball-cap (as a result difficult to guess his age)
**M3, caucasian, mid-late 20s
**M4, caucasian, late 20s - early 30s, stubble
**M5, caucasian, early-mid 30s, wearing parka,
**F1, caucasian, early-mid 30s, left the store with M5 - a couple?
**M6, asian, late 20s - early 30s, blue sweater,
**F2, (my partner), caucasian, mid-20s
**M7, caucasian, overweight, wearing iPod earbuds
!!!Observations (roughly in order of occurrence):
*A rock record plays over the stores岥o system, (not anything familiar to me)
*ZF (at LP checkout counter) greets us as we enter west door into LP section
*M1 browses the ~DVDs on display in the Videomatica section
*M2, M3 are listening to ~LPs on turntables located on LP checkout counter, both wearing headphones
*ZF chats with M1 while she filed away some ~LPs in the LP section the store, ZF very soft-spoken, could not be heard over music playing on the store-sound system
*M4 browses the ~LPs in the recent releases section along the central wall, behind the LP checkout counter, has at least one LP in his hand as he browses
*Myself, F2 browse in the LP section, F2 leaves to browse the CD section
*A movie starts playing on the TV above the Videomatica checkout counter (MGM studio banner, the iconic roaring Lion, is playing), VM walks to LP counter briefly, then back to Videomatica counter where he remains
*The record is changed, another rock record, a saxophone solo is prominent
*M2 is joined by M1 at the LP checkout counter, purchases a small number of ~LPs from ZM, ZM fumbles with the receipt paper roll for the cash register, M1 and M2 leaves; ZM says ὦriendly manner to M1
*I enter the CD section of the store
*M5 browses the CD section at the back of the store, F1 looks at the contents of the music magazine, ~LPs on the display shelf, music-related books (is she bored?)
*F1 and M5 chats with ZM at CD checkout counter, first part of conversation could not be heard due to music playing over the storeﵮd system; when the side of the record playing finished, ZM could be heard relating story about crossing the border (the US border?), and 䠳eemed like a good idea at the time�[someone?] not being allowed into the country, afterwards, F1, M5 leave (did they buy anything?)
*M6 enters CD side of the store from the LP side of the store, approached CD checkout counter, purchased concert ticket from ZM (could not hear M6餥 of the interaction), bought 1 ticket for 30$ + 1$ surcharge for using a credit (debit?) card, leaves, ZM remains at CD counter
*The rock record finishes, ZF in LP section of the store
*ZM and ZF have a conversation (could not hear contents, at a distance)
*Myself, F2 browse the CD section, no other clientele present in the CD section (cannot see whoP section, if M3 or M4 are still there, but I saw M7 through entrance to west side), F2 wears headphones at listening booth along east wall of east room
*A classical record begins to play, a familiar piece (title and composer escape me), at what must be double the volume, myself, F2 surprised - (given my previous visits at Zulu, this is unusual, both in genre choice and volume, however I suspect it may not be unusual for this day and time in the evening, nearing closing time on a Friday night)
*F2 selects 2 ~CDs, purchases at CD checkout counter from ZM; we leave through east door
!!Reflection
In the record store (Part 1), I felt as though the act of observation disrupted my participant activity. In past visits to record stores, I often have a mental list of items that Iﯫing for, or a list of questions I might ask the (often knowledgeable) clerk(s). If I like what婮g played on the store䥲eo, I will ask the clerk to tell me about it, should it be unfamiliar. If it᭩liar and I like it, I often acknowledge the selection in conversation with the clerk or a friend. If I鴨 a friend and I find something interesting while browsing the shelves, I would call them over to show them or ask them about it.
I found that I was unable to do any of these usual activities as a participant-observer, due to a worry that I would miss observing something important. During my observation session, I acknowledged the music being played over the store䥲eo but didnﲭ an opinion on it. An exception to this occurred when the clerk began playing a classical record at a loud volume. I acknowledged this event as being unusual and atypical for the store, and I recall enjoying it.
I remember wanting to look for specific items, or browse specific sections of the store, but I couldn奭 to maintain focus on both this and my ongoing observation.
What I did find interesting as an observer was how much notice I took of the clientele. I realized that in my previous record store experiences, I would take notice of the clerk(s) and my friend(s). I likely wouldnᶥ noticed the putative father/son dyad or the couple (M5, F1). I had the expectation that record store clerks interact with shoppers and other clerks while shoppers interact only with clerks, they browse or listen in isolation. This is despite myself being an obvious counterexample: I acknowledge and interact with my friend(s) when in a record store. I hadn⥶iously taken notice of people who were shopping together and how they were interacting. I was able to observe interactions I hadnﴩced before, however at the cost of my own performance as a participant.
In the athletic centre (Part 2), the result was similar, but for different reasons. The athletic centre in question, the Richmond Olympic Oval, is an impressive structure with an enormous domed roof overhanging an amphitheater of multiple co-occurring athletic events. From my vantage point, I could observe 2 hockey games, a row of cardio machines, and basketball practice. This was in addition to spectator and staff activity, as well as the flow of people up and down the stairs and escalator behind me. My problem for the first 20-30 minutes of my observation was been unable to decide what to observe; there was an overabundance of activity. This time was also spent analyzing the layout of the facility. For this reason I chose to continue observation for an additional 20-30 minutes, but with a tighter focus on the activity surrounding the hockey games, rather than activity on the basketball courts or in the cardio machine area.
Despite a narrower focus at the end of my observation session, I missed several important events (such as goals scored in one of the hockey games). In fact, I didn note of any pertinent details of the actual game play on either rink: whether the players could skate well, the level of physical contact between players, the overall level of talent on the teams, or who the key players were. If I recall previous hockey games Iᴴended as a spectator, I expect that I would have noticed these details. On the other hand, I found it odd that despite roughly two dozen spectators present for one of the games, there was no obvious cheering or applause when a goal was scored, which would have caught my attention.
As a participant, I found myself scrambling to record as much as possible, writing constantly for the duration of the observation session. Thus I spent much of the time hunched over my journal, which rested on the bench where I sat (resulting in a sore back and hand).
While I didn奬 uncomfortable taking written notes in a public place, to an outside observer I must have appeared to be oblivious to the events that surrounded me, behaving quite unlike the other spectators. I likely appeared as a student working feverishly to finish up some homework, looking up from my notes with an unfocused stare when writer쯣k struck.
In summary, my observation got in the way of my participating in both the record store and at the athletic centre. In the former case, I didn field notes in situ, and as a result I noticed events external to me, but I didn㫮owledge my internal reactions and intentions. In the latter case, my field-note recording got in the way of my observation. Not only was I unable to truly participate as an spectator of a hockey game, I was unable to truly observe anything due to the preoccupation of taking notes.
!!Evaluation and points for improvement
Know (roughly) what to expect. I went to the athletic centre without knowing anything about the layout and size of the space, the type of activities occurring, the proximity of these activities to one another, and what might be occurring there at 10pm on a Wednesday night. A few minutes of searching the web would have revealed all of this information. I decided to carry out my observation there without any prior knowledge, as a blank slate, with no more than the expectation that I would watch a game of one sport or another. When I arrived, I was confronted with too much information and couldn壩de what to focus on.
Reconstruct the context afterwards. I spent too much time surveying and noting details of the layout of the athletic centre (as a result of not knowing what to expect). I took 3 pictures. In hindsight, I should have taken more photos, or I could have shot a short video with sound using my phone or iPod. From this material I could reconstruct details of the context later. In the record store, my familiarity with the context allowed me to reconstruct details of the context easily afterwards.
Record my own perspective and actions as a participant, be more reflective. Observation was too all-consuming in both settings. I should have considered and reported my own impressions of the music being played over the record store䥲eo. I could have reported the interesting items I came across while browsing. I could have chatted with the clerk(s), as I have done on other occasions and reported what we chatted about. It is also possible that in writing my observation notes post-hoc, the descriptive observations were retained while reflective thoughts were lost. The next time I write notes post-hoc, I should record reflective thoughts before descriptive observations, while they are fresher in memory.
In the athletic centre, while I spent a significant amount of time recording descriptive observations, I could have taken more time to reflect: to appreciate the talent of the hockey players, or to predict whether or not the trailing team would make a comeback. In short, I should report my own feelings, reactions, predictions, and inferences to a greater extent.
Develop a better shorthand. I spent too much time in the athletic centre writing notes. Should I know what activity to expect beforehand, as well as the layout of the context, I could develop an a priori code set for anticipated activity, persons, places, and objects. This will allow me to spend more time observing and less time writing (also sparing my hand and back). Taking photos, video, and voice memos would complement a better shorthand.
!!Topic: Conception of learning:
//Prompt//: What does learning mean to the interviewee? Research tells us that people sometimes think about learning as change, interpretation, abstracting meaning, acquiring facts or skills, memorizing, increasing information.
!!Part 1a: Initial Questions
#Last week I was editing a colleagueﵲnal paper draft, which was about the learnability of technology designed for use by older adults. What was written got me wondering about how we describe the process of learning. Specifically, I wondered what the difference is between statements like these:
##쥡rned to use a computer�earned how to use a computer襠learned to write effectively�quot;she learned how to write effectively堳elected a set of images (below) that I feel address, in some way or another, a process and a context of learning. With regards to learning, what comes to mind when you look at these images?
##wholved in the learning?
##how are these [processes/contexts] [similar/different]?
#Whatxample, either from the preceding set of images or from your own experience, of 䳭on learning㣈ow is this different from other forms of learning?
#Some people self-describe as 砬earnersﭥtimes they are labelled as such by others. In both cases, how do you think this is determined?
##How do you think someone feels when this label is imposed on them?
[img(50%, )[http://cs.ubc.ca/~brehmer/research/595w3.png]]
!!Part 1b: Rationale for initial questions
#This �p�ion is structural (and somewhat playful), grounded in my experience, giving the interviewee an indication that the question is personal to the interviewer, that I am genuinely curious about the topic that weut to discuss. It would be somewhat more abstract of a question had I decided not to ground it in my experience. My aim is to understand the interviewee岳onal conception of the structure of learning, whether the statements are equivalent. If not, is ⮩ng how to�essary part of ⮩ng to�ning how to�of a sequence of ⮩ng to㔨is question uses a set of 9 images (above) as an elicitation device. By providing several concrete reference points, I again address the descriptive and structural aspects of learning. In some instances, the social aspects of learning are also addressed. Some of these instances may also elicit opinions and feelings drawn from the interviewee谥rience in different contexts, so the question is also personal and cathected.
#This question is a descriptive 鶥age question. However it also addresses the interviewee岳onal opinion and conceptions regarding the term. By giving the option of choosing from the preceding examples and their own experience, the question can be personal, should the interviewee feel comfortable relating these experiences.
#This question has a similar rationale to the preceding question, addressing another 鶥 term襠subquestion is indirect, personal, concrete, and cathected.
!!Transcript
//Friday, March 9th, 2012, ~3:15pm, Bean Around the World (coffee shop)//
//[primary handheld recording device failed, using backup recording (from laptop, not as reliable) until 0:14:38, [...] = inaudible]//
0:00:17.2 M: So lets start by asking about something that came up in my experience last week. I was reviewing a... a... uh... journal article draft by a colleague of mine in my department, and her article was about the learnability of technology [...] for older adults, and throughout the article it kept making this statement about "learning to use", and for some reason that bothered me. I thought "learning how to use" a computer, for instance, or "learning how to use" something was more representative of what she was talking about. So I was going to show you these two sentences. ...Um "he learns to use a computer" and "he learns how to use a computer". So I was wondering what you thought the difference was between those two sentences.
0:01:15.5 J: well I think operatives there is the word "how" which basically then denotes to me that there's a process that is taking place and therefore I could ask and probe more about well what exactly and how [...]. So you can actually scaffold understanding of learning processes. Whereas the other one is "to learn", we don't know whether he learned everything [...] like the process is implicit but we don't exactly know that he used it in what way specifically. So that's my [sense].
0:01:57.4 M: So there's this uncertainty about whether or not the person understood the mechanism of how they use a computer versus they just knew [..] when or why they should use a computer. Is that what you're saying?
0:02:16.8 J: I think so. I think the function that - I mean computers are meant to be functional and to assist us in several tasks so learning "how" to use a computer is more [...] the computer is to be used for a task. Whereas they learned about computers it could have been that it's just learning it from a book. They didn't actually go to a class [..] hands-on application of knowledge.
0:02:49.6 M: So that's [...] I'm going to jump ahead a bit the dimension depends on sorta coming back to this, this question: what is, or what do you think comes to mind when you say "hands-on learning"?
0:02:59.6 J: Hands-on learning for me is when you're actually applying current uhh current information that you know of and you apply it to new information so you are accommodating and assimilating information but you are holding it in to your own understanding [...] or in a way that makes [...] but in a process. So you're not not just learning about it technically speaking from a book but you're actually [...] engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body. Like sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember. So I have a bio-kinetic memory of what my fingers are doing in order to help [me guess] the password so I think we learn, at least in my case I think when we learn we learn with our whole body and we just don't [...].
0:04:09.1 M: So by that understanding how something works, so learning how something works isn't "hands on"? It's the actual doing, it's the learning "to"? Is that what you mean?
0:04:21.7 J: Well, connecting it back to your question you said about what the [...] question of how we learn. [...] Maybe there is some procedure that means that [...] interface with that machine, or whatever, computer. But what I mean by that is that [...] about the how doesn't necessarily make it clear to me that the person has used a computer in a way that [...] functions. If it's functional then it means that you're actually having to engage in hands-on, you're actually using it rather than just theoretically [..]
0:05:15.7 M: It turns it from something abstract to something more concrete. Is that what you're saying?
0:05:22.3 J: Yeah. I think so. Yeah.
0:05:25.3 M: My second example. Perhaps it's different, maybe it's the same, maybe it's different. Is um, the example of "she learns to write effectively" and "she learns how to write effectively". So is that the same or is that different?
0:05:40.7 J: I think it's the same. I think I would define it and operationalize it. I think that learning how to write versus [...] she thought of [...] she had to write papers, whereas the other one is more like she takes a grammar book and she just kinda learns rules [....]. I think knowledge to me is theoretical or applied but when you apply it becomes something more like it's integrated into you.
0:06:14.8 M: So if your goal is to apply knowledge, is it necessary to understand the theoretical side of it as well?
0:06:24.5 J: Absolutely, but when I mean applied I mean you have the theoretical and then you learn how to apply the theoretical so when you apply the theoretical you actually learn how to [...] yourself, that interface, that product, that machine, computer, even the [...]. Uhm.
0:06:51.7 M: So by counterexample, yer, you can write well, or you can write effectively but when asked about the rules of grammar you're employing, or the rules of sentence structure, and you have a hard time describing that. Is there a disconnect there? In understanding the "how" and ...?
0:07:12.8 J: That's a good question because that to me is a good example of having actually applied knowledge to the point where you can [...] It's a really nice distillation of knowledge where you have learned how to [...] in a way that makes sense to you and you don't have to be explicitly thinking about it, it's become an implicit process. And you can then further scaffold more knowledge on to that. [...]
0:07:57.4 M: What I'm going to do now - that was my warmup question, it was actually serving to my own purposes because I was genuinely curious about what the difference was between learning how and learning to. I'm going to jump into my main question. [It's uh, is this still working, ...ah, here it is.]
0:08:25.0 J: I had a bit of an addendum to the [...] how [...] it's like a top-down, as if you have to do it versus learning how something [..garbled..]
0:08:38.1 M: So I've selected a set of images that I feel address I think in some way or another uh either a process or a context of learning. So with regards to learning I am wondering what comes to mind when you look at these images. [J: What comes to mind?] You don't have to address these in any particular order, I'm just curious what your reaction is with regards to learning in any of these nine images.
(J observes images)
0:09:20.7 J: Well the first thing that comes to my mind is that each one is ... in a context. [...] In a framework, [...] that is in itself a context. But each one is [...], even though some are done in [...] a very white background. Um so they have both a ... story that in that I have a current understanding about what [...] including some where there is just one person sitting [....]. So the top left, that is uh a stereotypical example of learning where we're assuming that older people know more than younger people. Um, perhaps he's a driving [...] proved experience, more years driving. Uh but that's not necessarily the case, you got a person that's never driven, I mean for all I know I'm making an assumption here - I'm thinking that's a driving test, or driving lesson, but it could be a person who's never driven, late for a meeting, and his daughter is driving him, and she can drive.
0:11:00.5 M: In that latter case what would the context of learning be?
0:11:04.7 J: So In that case the context of learning is that he is probably as we're seeing a different kind of learning, a social type of learning where she has learned to help her dad, [...] so yeah, there isn't just contextual learning but there's relational learning so [...] when he needs a ride you';ll give him a ride. Um similarly with the next one to the right where there is an older person teaching some younger people about what looks like chemistry, so in that case in seems like it's [...] where chemistry is somewhat of an advance abstract abstract knowledge that requires some formal orientation as to what to do. So that would definitely apply a "how". [...] Just going back to the first picture, that one also has a how component in it, it in is a picture of a man teaching here "how to" drive, "how to" learn about. And the image to the right of the little, is it uhh, a stuffed animal, by himself, but he also has a context in that [...] inanimate object, the context is dependent on the [...] wants to apply to him or her or it. So I don't know if there's any learning going on there per se, but...
0:12:54.8 M: Perhaps I'll clarify that. Um. This is a very popular children's toy in the late 90s and early 2000s, it's called a Furby and the big selling point about this toy was that it had the claim that it could talk and that it could learn to speak English. It could learn your name. And it could speak back to you. It could learn certain words.
0:13:19.5 J : Oh well in that case maybe you're a computer guy maybe you're trying to get me to think about whether the machine, that has a computer chip, can learn. In that case no I don't think so. In that case the machine has actually captured knowledge but [...], so it's like a mirror. [...] So I don't think that that this person is [...] that Furby can see. Uhm, there's developmental differences perhaps if you gave that to a two- or three-year-old and told them it could speak, that Furby can speak, they may believe that they can teach Furby to speak but that's not the case. Um however relationship for [...] a four-year-old, five-year-old depending on how [...] they are you could be a tool that is a social [...] tool that will help a child learn how to interface with another pretend [...] so it shows them communication... transfer of knowledge.
0:14:38.1 [switched to other (handheld) recording device here].
0:15:00.0 J: And then um the gentleman with the lovely uh dog I think um he that's I think that my personal view is that humans learn as much from pets as pets learn from humans so I think that the knowledge there is bidirectional for sure, equally. Then they just have to know common symbols just as you and I know common symbols in language. I think in this case the guy is showing the dog a hand signal, the dog knows "ok, that means something, I'm supposed to sit" but um the dog similarly probably in their interaction in in [...] the man what the signs are for "I have to go to the washroom now if you don't want me to pee in the car or on the carpet" uh yeah. An then the one in the middle uh.
0:16:00.1 M: Let me just interrupt for a moment if I can. So that I understand. You mentioned that learning thus far is a property of animate objects, it's not a property of inanimate objects, being the Furby up here. The dog can learn but a toy cannot?
0:16:19.8 J: Correct.
0:16:23.1 M: And how can, how is it different from what the dog can learn and what the human can learn?
0:16:33.0 J: I think the difference is qualitative, the difference is that the dog uhhh physiologically does not have the same organism, the same brain, and I think that's relevant to your computer analysis uhh whether computers are actual brains or not or simulations of, but the main difference is that it's two different species and so they need to find a common language and so that is, that's how I would generally answer that question.
0:17:11.8 M: You can continue if you want or if you have anything else to add, [J: You mean the other pictures?] about the other pictures.
0:17:16.4 J: I can go through all of them if you like. [M: Sure, if you feel like it's]. Are we done with the man and the dog one? [M: Yeah, you can continue] I can continue on that one? [M: You can continue with the other images] With the other ones, OK. Ok so then the uhh the one in the middle with the two guys again there's an age difference so I'm thinking there's an assumption that but not necessarily the guy that looks a little bit older he could have started school later and it's actually that he's the, the engineer in his practicum and that the younger guy could already be a P.Eng so that's all the assumptions that we bring to these kinds of pictures but um learning there I think I don't know is he handing him something? [M: So this is a picture of of a man with a laptop computer - oh it's here?] Sorry this one. He's handing him something, is it a camera or a cell phone? [M: Uh, I'm not sure]. Anyway, that but ok that's one I was but then again that's also it's the way they're dressed implies a trade so something they've learned about the trade that's why they're involved in that particular trade and so again they have a common language that they have to learn in order to work in that industry. So I think that we're feeling now that I'm thinking about what I'm saying we're discovering a particular, in order to do a particular task there needs to be a common language. And that the only difference is that we've covered the Furby that one's inanimate so that one's not, we're not uh, we're transferring knowledge to it but it's only a knowledge repository that is the same kinda knowledge we gave to it it's not expanding it or helping it grow it's just repeating it back to us. And it might have a computer chip that helps it have an algorithm that manipulates grammar and things like that but again and that's been transferred, downloaded or transferred over from a human, and its, yeah, it's following an algorithm.
0:19:22.2 M: So the child speaking to the toy, what common language is there between the child and the toy?
0:19:28.3 J : There isn't. Because the Furby is not a sentient being.
0:19:32.7 M: But the child is learning about how to communicate with others. How is it that the child can learn to communicate with others when the other is not reciprocating?
0:19:46.2 J: I think in some ways the child has a chance to embody a practice more than anything practice what they themselves have learned with their actual sentient beings in social relationships and they can practice it because if they have busy parents, busy environments, they can spontaneously approach a very available inanimate object that's there any time.
0:20:15.2 M: So they're practicing this shared language that they have accumulated with other adults or other individuals that are sentient presumably, and this toy is just used to further that process of learning, that practice?
0:20:15.2 J: Yeah. It's more like a tool. It could function as a tool. At least I'm trying to see some usefulness rather than it being a space-taker on my shelf.
0:20:45.4 M: Anything else, any other reactions to the other images?
0:20:50.2 J: Well I guess that one that you say, the gentleman that I uh maybe he's in his 70s perhaps I have no idea and he's in front of a laptop. I don't know, I think if he's interacting with a computer in that way I'm thinking that he's already learned how to use it, so he's learned, and he's using it as a tool to communicate with others. An then the children at the bottom I think it's peer-to-peer learning, which is also very helpful and it's similar I guess to interacting with the Furby but in this case there really is knowledge construction amongst peers ummm amongst all people engaged. In the case of the Furby there could be knowledge in the sense that the child as it's interacting with it may learn some skills um but there's only, it's just the child learning, the Furby could not. And then.
0:21:49.1 M: Whereas with the other children there's an opportunity for this sharing of learning?
0:21:54.0 J: Yeah they're sharing, they're sharing and there's also different opinions, different perspectives that are key to construction of knowledge. And then the next image in the middle is I guess the Dragon Naturally Speaking device that I have on my machine here that we are working with so some ways is also a tool for learning between us in our interview and uhm developing skills I suppose. Uhm yeah it's a communication device.
0:22:25.6 M: But what about the claim that it can learn to tell your voice from mine? And that it can learn to detect the words, perhaps if you have an accent, or if you have a certain way of speaking it can detect your way of speaking but not mine. So I included this because it's similar to the Furby in the sense that there's claims around these technologies that they are able to learn and that is what is enticing about them.
0:22:55.7 J: Well I don't necessarily agree that they're learning at all, I think that the reason why they're able to capture certain features about human knowledge is because we program them with that knowledge and we have studied say like on sound if you analyse sound you have to break it down into frequency you break it down into tone, so that's knowledge that we have and we have taken time to study and analyze the detail of the language and communication and then we are looking for devices that will facilitate that kind of uh medium of communication. So no I don't think that Dragon Naturally Speaking and or the Furby are learning they are simply tools like a medium that are an extension of our current knowledge and that we're using to build further knowledge but they're not learning.
0:23:56.2 M: So.
0:24:05.6 M: If I understand correctly that we have learned how to understand each other speaking or understand uh what is meant by our speech and/or how to detect differences in accents or detect differences in uh pronunciation or ways of speaking and we've just passed that knowledge on to these machines and the machines are not then learning it themselves they're an extension of we've done.
0:24:35.1 J: Yes. And I wouldn't even say that we're passing knowledge on to them, we are, we are, we deconstruct our knowledge to mechanical aspects that are the may layers of what's the act of what is engaging you and me right now which is language and communication which is as you know anthropologically on and on that's what differentiated us ages ago other species and even amongst the many human-like forms uh that were coming up like the famous Neanderthal, that's right, even though they were more populous somehow the, we, our species won the race. So yeah I just think they're devices. And then moving on to the young lady sitting in the Forestry building it looks like, I don't know where that is but it looks like the Forestry, I think it's a library that looks like the Forestry building [M: It does, yeah]. I think it's active learning which she's picking up from a book but that she'll be expected to apply either in her understanding in a midterm or in a paper.
0:25:49.0 M: So how is that different from say the other students, presumably these are students, in this chemistry lab setting, in the image at the top.
0:25:57.9 J: Well personally I think that active learning, there's some, I'm thinking that that is a chem lab, and they needed to do the same thing that the woman at the bottom in the library is doing, eventually you do have to sit with a book, eventually you do have to know the techniques, eventually you do need to know that if you mix this and that it's not a good idea because it's toxic, a fume could come up and you're out like a light. So yeah, you have to integrate both, like the as we were talking earlier, you learn about the theoretical and then you have to put it into practice and I think it's a feedback loop, talking in computer terms I suppose, yeah.
0:26:35.5 M: So these processes alternate?
0:26:36.7 J: I think they co-exist. But I guess temporally if you're looking at it as a computer programmer yeah they would have to they alternate. Yeah.
0:26:52.4 M: Are there any other ways that some of these images or pairs of these images are similar?
0:26:58.6 J: Similar? In terms of learning, I think some of them are showing processes of learning, some of them are showing bidirectional learning and some of them are showing devices that can facilitate learning.
0:27:17.5 M: For the relationships that are bidirectional, I think you indicated before that there was a certain, it was bidirectional but there was a, I'm searching for a word... bidirectional but imbalanced. Perhaps there was someone who was doing, who was more senior and someone who was more junior.
0:27:56.5 J: There I think I'm talking more about an assumption that learning happens in that one-directional and in that downstream kinda direction. But I definitely think that's an assumption and I think it's a stereotype, and I don't think it's a good stereotype, because I think uh many people that I know they keep saying that the most that they've learned in their life is from their children more than anything so I mean, but there's different kinds of knowledge, right, there's the textbook knowledge where like you do have to go hit the books, there's like the chemistry photo there that eventually you do need to get somebody that's more experienced to teach you actually how to go through the procedure, they're showing you the hands-on component that you will also need to go through and as they're teaching you this, with that particular picture, they, the guy, it's the teacher who has their hands on the equipment and is teaching them how to use it even though they probably read the protocol as to what to do for that particular lab. Yeah.
0:28:58.4 M: So what is, if that's the chemistry teacher, what is he learning from his students?
0:29:03.7 J: At that moment, is he learning from his students, is um, I don't know, in that case that he's learning about the same thing as the students are, like how to use the equipment because it's something he knows already but I think what he's learning at that time at that moment that students are in at different levels for example the student on the right and the student on the left are different people, they might have different kinds of questions, so as they're interacting he's learning about how the girl on the right is doing and how the girl on the left is doing and what level of processing they each have so I think he's learning about his students and maybe he's learning about how he could better prepare his lecture for the next year or to make it more clear or whether he at end of this particular lab he might want to tell one of the women "well, come and see me after class if you want to expand on that or if you're really interested in that molecule that we talked about I have some really cool papers that you could read". So he's learning about both of them that way.
0:30:20.5 M: And how is that similar or different to say, the image underneath it, in which you have, again bidirectional learning, presumably uh, you mentioned before that this indicates a trade, learning a trade. What is the difference between, you mention up here is that they were learning about how to apply something theoretical like the woman down in this corner, it implies that they've already done some book learning, some theoretical learning, and this is their step to apply it. How is it different from learning this trade?
0:30:55.7 J: Well I guess because we are looking at pictures and we can only go by what we're seeing in many ways the two guys that are wearing the, even I there am making assumptions, thinking it's a trade, for all I know they're both P.Eng's already and they just have to go in and have a photo op. Um. Or they could be the guy that goes in and lays the bricks. But the learning process, that particular picture doesn't necessarily mean that they're learning from each other, it looks like a photo op. That's it, but because you're asking me about learning I was pushing it a little bit further. Yeah.
0:31:36.9 M: So my earlier question about hands-on vs. hands-off, do you think that's fairly well represented here? Or if there's something that's missing? What do you think might be missing of what's capturing the many processes of learning?
0:31:58.1 J: Well the basic thing is that there are two um, there's, it's always dialectic, period. That's what I would say. Ha ha. In each one of these uhh in order for the communication to happen in order for the learning to happen it has to be a dialectic process so even though the Furby at the top is in the picture by itself uh in order for it to work it needs to be actually used or touched or programmed or spoken to or yeah, it has to be a dialectic process, yeah that's what I would see as a commonality. But what I would see missing, hmm, not necessarily, they're all each different contexts, but they uhh, no I don't necessarily see anything that's specifically missing.
0:33:05.7 - 0:50:48.9 [discussion of the term "slow learner", individual learning styles]
0:50:48.9 M: I think that's. [J: Is that it?] I think that'll have to do for now just otherwise it'll be 17 hours to transcribe. [J: Ok - oh look, let's cross fingers. Ok ..[...] oh dear. Ha ha] Apparently that was 50 minutes long [J: Five Zero?]. Five Zero. [J: What are you doing, it's supposed to be half an hour?! Ha ha ha]. I may only take half an hour of it. We'll have to see I may only half an hour.
0:51:34.2 [end].
!!Reflection
Review the transcript and reflect on how you might improve your interviewing skills. Consider, for example, alternative questions you might ask; missed opportunities and what you might in retrospect have done; the level of rapport you established, the quality of your questions, and so on.
//Listening//: During the interview, I noticed my habit of thinking of my next question while the interviewee was still speaking. I had to internally remind myself to listen, to delay thinking about the next question I was going to ask. These reminders were a distraction. Our time constraints and environment were also distracting. Many of these reminders were along the lines of ᤠbetter make the best of the time that I have and the place that we鮬 so I崴er ask good questions that will elicit rich data-rich responses, and I shouldn쬯w any dead air time奠to these distractions, many of my questions were confirmation questions. To be realistic, many interviews will be conducted under time constraints and held in distracting environments, both will be difficult to ignore. However, I shouldn場oo afraid of some dead air time. I should concentrate on listening and use the dead air time to process the intervieweeﲤs and develop follow-up questions.
//Confirmation questions and paraphrasing//: many of my questions were of this form. 䨡t what you're saying?�:57, 0:05:15.7) and 䨡t what you mean?�:09.1) were asked following a paraphrased statement. This is a habit of mine that Iiced in past interviews I㯮ducted. The fact that this type of question only appeared near the beginning of the interview suggests that I may have caught myself doing this and avoided repeating this habit during the remainder of the interview. However, it쳯 evident that there are very few clarifying questions as an alternative. Most of the questions that werenﮦirmatory were detail-oriented, probing questions. Some were yes/no questions that elicited short responses (e.g. यg can learn but a toy cannot?�00.1). Fortunately, I didn㫠any �stions. In future interviews, I must remind myself to ask clarifying questions, avoiding confirmatory and/or yes/no questions.
//Multiple questions//: I should limit asking multiple questions in the future. An example appears at 0:31:36.9: �earlier question about hands-on vs. hands-off, do you think that's fairly well represented here? Or if there's something that's missing? What do you think might be missing of what's capturing the many processes of learning?࣯uld have simplified this to the first question only, asking the additional questions if not addressed in the initial response.
//Rapport building//: My interview partner began her interview with light smalltalk, asking me how my day was going. I, on the other hand, began my interview by asking a detailed and concrete question (0:00:17.2) that reflected my personal experience. Initially I thought this could be a fun experiment and a way to signal my personal interest in the topic at hand, but to some extent I think this strategy backfired. This question didn쩣it a personal response from the interviewee, and did little to build rapport. There is a balance between an initial question that 峠right in�e details, as opposed to an abstract, high-level 10,00ft question, and one that establishes personal rapport. Perhaps it is as simple as prefacing the initial detail-oriented question with smalltalk. Alternatively, I could have asked: 젭e about your early experiences using computers�the follow-up question: 줠you say that you learned to use a computer or that you learned how to use a computer?詳 would have been more personal, and yet still concrete and detail-oriented.
//Elicitation device//: The diorama of images I used generated a lot of discussion (0:08:38.1 - 0:33:05.7). As a result, I was unable to ask to my final initial question regarding the term 砬earner�e first half-hour of the interview. Nevertheless, I was glad that it generated so much discussion. However, responses were not very personal and somewhat abstract, as the images did not elicit any personal experiences that corresponded to any of the scenes depicted. In hindsight, I should have made these scenes personal by asking 堹ou ever been in a scene like this?�ach image (e.g. 堹ou ever taken driving lessons?� you ever taught a dog how to behave?�you ever taken a laboratory-based chemistry course?�ernatively, some of these questions could have been indirectly personal: 鯵 have an older family member or friend who has learned to use a computer later in life?�ou know someone who works in a trade, or has been an apprentice?ꁮother strategy I could have used would have allowed the interviewee to take personal ownership over these images: I could have printed this images on index-card-sized sheets and asked her to sort the images. This could have generated a substantial amount of discussion based on the interviewee岣eived similarities and differences with regards to learning depicted in these images.
//Interruptions//: I feel as though I successfully limited interruptions. I did knowingly interrupt once (0:16:00.1) in order to explain what a Furby was and the claims surrounding it, as it appeared as though my interviewee was unfamiliar with the toy (and the response to it hinged heavily upon its characteristics). I also allowed my interviewee to interrupt when she had an addendum to the previous question (0:08:25.0). Unfortunately, ambient noise made this addendum difficult to interpret and transcribe.
//Technology issues//: we attempted to use 3 recording techniques: a voice memo application on an iPod Nano, the ~ExpressScribe application running on a laptop, and a Dragon Naturally Speaking Dictation application on an iPhone. Unfortunately we were in an environment with a significant amount of ambient noise (one might expect that a late Friday afternoon at a campus coffee shop might be less busy, but this was not the case!). For this reason, the Dragon Dictation application did not record anything meaningful, only grabbing a few words here and there. Just prior to beginning the interview, we were interrupted by an acquaintance of my interviewee. Following this, I forgot to un-pause the recording on the iPod (this was corrected at 0:14:38). We relied on the laptop壯rding until this time, which was of poor quality. Noise removal in an audio post-processing application resulted in a marginal improvement in fidelity. In the future, I will find a quieter interview location ahead of time, at least when possible. I will also record, process, and listen to short test recordings before I begin the interview. In the case of interviews where the location cannot be changed or when interview context matters, ambient noise may be unavoidable. In such cases, a directional microphone may be a good investment.
When it came to transcription software, I used the free version of ~ExpressScribe. The ability to create custom keyboard shortcuts for audio playback was helpful, however my praise stops here. It appeared to have no auto-save preference, no obvious and visible save function, and what could be a bug in which toggling to another audio file results in a loss of the dictation text from the previous audio file, with no way of recovering it. As a result, I lost about an hourﲴh of transcription at one time. This loss of data, in addition to the poor audio quality of the laptop recording, made transcription an agonizingly slow affair. My transcription-time to interview-time ratio was at one point 4 hours for 15 minutes, or 16:1. Before transcribing another interview, I will find (and likely purchase) a superior transcription application.
//Transcription shorthand//: As I develop my skills as a transcriber, I will develop a transcription shorthand similar to that presented in David Silverman`Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research. By comparison, my current transcription doesn榥ctively capture nonverbal communication, volume or intonation of speech, or the length of pauses - these, if at all, may have been recorded in an ad hoc fashion in this transcription.
[img(90%, )[http://cs.ubc.ca/~brehmer/research/595w4_1.png]]
[img(90%, )[http://cs.ubc.ca/~brehmer/research/595w4_2.png]]
!!Part 1
//Choose a small piece of data (not more than 10 lines of text) from the [[interviewing workbook|EPSE 595 Workbook 3]] to analyze. Look at a micro level and develop characteristics that might be the first step in an analysis of your complete data set. Include the text you are analyzing and the analysis.//
!!!Text:
> J: Hands-on learning for me is when you're actually applying current uhh current information that you know of and you apply it to new information so you are accommodating and assimilating information but you are holding it in to your own understanding [...] or in a way that makes [...] but in a process.
>So you're not not just learning about it technically speaking from a book but you're actually [...] engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body.
>Like sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember.
>So I have a bio-kinetic memory of what my fingers are doing in order to help [me guess] the password so I think we learn, at least in my case I think when we learn we learn with our whole body and we just don't [...].
!!!Preliminary [micro-level] analysis:
!!!!Code generation process:
*''1st iteration'': generated word frequency* list (not including stop words, common words), generated Wordle (below) to get overview of word frequency. 9 of 43 words repeated: learn (6), actually (3), information (3), body (2) current (2), just (2), password (2), think (2), and whole (2).
[img(50%, )[http://cs.ubc.ca/~brehmer/research/595w51.png]]
*''2nd iteration'': collapsed words with similar meanings into preliminary codes, generated a new count:
[img(50%, )[http://cs.ubc.ca/~brehmer/research/595w52.png]]
*''3rd iteration'': used preliminary codes in ~HyperRESEARCH
*''4th iteration'': created code groups: 3 semantic (lumping related terms together: cognitive activity, relating to a sequence, specific objects), 1 structural group *
*''5th iteration'': 2nd coding pass, added additional structural codes; created meta-level structure code group (semantic and structural, reflecting 4 sentences / short paragraphs: idea 1, idea 2, example, summary.
*''Result'': 24 unique codes, 4 codes groups, 1 sub-group, 2 codes not in a group; 67 total code instances.
!!!!Resulting code list:
*cognitive activity (14, group)
**1. know (3)
**2. learn (6)
**3. memory (5)
*4. information (3)
*5. relating to the body (4)
*relating to a sequence / event occurring in time (11, group) *
**6. action (3)
**7. before and after (2) *
**8. process (4)
**9. transfer (2)
*specific objects (4, group)
**10. book (1)
**11. home (1)
**12. password (2)
*structural (27, group)
**13. negation (2) *
**POV (8, sub-group)
***14. 1st person (6)
***15. 2nd person (2)
**16. specification: irregular (1)
**17. specification: definitive (5)
**18. indigenous typology (4) *
**19. repetition (4) *
**20. transition (2) *
*meta-level structure (3, group)
**21. idea 1: transfer of knowledge (1)
**22. idea 2: involvement of the body (2)
**23. example (1)
**24. summary (1)
!!!!Codes / key-words in context, with annotation and code descriptions:
!!!!!Code group: cognitive activity
''Frequency'': 14
!!!!!Code 1: know
''Frequency'': 3
''Description'': understand / know / think [believe]
>holding it in to your own understanding
''MB Annotation'': holding = retaining, read as "comparing it to you own understanding"
>I think when we learn we learn with our whole body
''MB Annotation'': I think = I know / I believe
>current information that you know
''MB Annotation'': You can know information
!!!!!Code 2: learn
''Frequency'': 6
''Description'': learning / assimilation
>Hands-on learning for me is when you're actually applying current uhh current information that you know of and you apply it to new information
>you are accommodating and assimilating information but you are holding it in to your own understanding [...] or in a way that makes [...] but in a process
''MB Annotation'': assimilating = learning / taking in / absorbing
>So you're not not just learning about it technically speaking from a book
''MB Annotation'': infers that learning can occur through technical reading in books, and also through other means
>you're actually [...] engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body
''MB Annotation'': learning is an act one engages in, infers a beginning and an end to the act
>when we learn we learn with our whole body
>act of learning
!!!!!Code 3: memory
''Frequency'': 5
''Description'': memory / forget / remember / hold [retain] / guess [recall], check [verify/compare] / help [prime / cue]
>you are accommodating and assimilating information but you are holding it in to your own understanding
''MB Annotation'': holding = retaining (in memory)
>sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember
''MB Annotation'': forgetting and remembering, complimentary ideas
>I have a bio-kinetic memory of what my fingers are doing
''MB Annotation'': memory can be physical / have a physical embodiment
>memory of what my fingers are doing in order to help [me guess] the password
''MB Annotation'': guess ~ recall / use partial knowledge
>I have to pretend like I'm actually checking in at home so that I also remember
''MB Annotation'': check = verify / compare with existing knowledge
!!!!!Code 4: information (not in a group)
''Frequency'': 3
>accommodating and assimilating information
''MB Annotation'': you can adapt to / accommodate information, you can assimilate / learn / take in information
>applying current uhh current information
''MB Annotation'': you can use information in an application
>apply it to new information
''MB Annotation'': information can be the object of application
!!!!!Code 5: relating to the body (not in a group)
''Frequency'': 4
>Hands-on learning
''MB Annotation'': hands / hands-on = part of body
>engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body
>So I have a bio-kinetic memory of what my fingers are doing in order to help [me guess] the password
>we learn with our whole body
!!!!!Code group: relating to a sequence / event occurring in time *
''Frequency'': 11
!!!!!Code 6: action
''Frequency'': 3
''Description'': act / apply / engage / make
>applying current uhh current information that you know of and you apply it to new information
''MB Annotation'': action = applying
>engaged in the act of learning
''MB Annotation'': action = engaged in an act = active ~ actively learning
>a way that makes [...] but in a process
''MB Annotation'': read: "a way that forms a process?"; action = forming/making
!!!!!Code 7: before and after *
''Frequency'': 2
''Description'': current [old / existing / prior] / new
>applying current uhh current information
''MB Annotation'': before: "current" read as "existing / prior", currently held information
>new information
''MB Annotation'': after: new
!!!!!Code 8: process
''Frequency'': 4
''Description'': process / path / way / in order to [sequence], case [instance]
>accommodating and assimilating information but you are holding it in to your own understanding [...] or in a way that makes [...] but in a process
>learning about the path of uh... biodynamics, bio-kinesis, like you're whole body
''MB Annotation'': path = process
>memory of what my fingers are doing in order to help [me guess] the password
''MB Annotation'': in order to = sequence / process / a step-wise action
>in my case I think when we learn we learn with our whole body
''MB Annotation'': in my case = my instance, my process
!!!!!Code 9: transfer
''Frequency'': 2
''Description'': accommodation [adaptation], pretend [simulate/emulate]
>applying current uhh current information that you know of and you apply it to new information so you are accommodating
''MB Annotation'': transfer (abstract) from current (read: prior / existing) information to new information
>Like sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember
''MB Annotation'': transfer (precise) from prior experience (at home) >current experience (pretend = simulate / emulate)
!!!!!Code group: specific object
''Frequency'': 4
!!!!!Code 10: book
''Frequency'': 1
>not just learning about it technically speaking from a book
!!!!!Code 11: home
''Frequency'': 1
>I have to pretend like I'm actually checking in at home so that I also remember
''MB Annotation'': remembering has a contextual component; where was the process initially learned?
!!!!!Code 12: password
''Frequency'': 2
>I forget my password
''MB Annotation'': password = object of forgetting
>to help [me guess] the password
''MB Annotation'': password = object of remembering
!!!!!Code group: structural
''Frequency'': 27
!!!!!Code 13 : negation *
''Frequency'': 2
>you're not not just learning about it technically speaking from a book
''MB Annotation'': hands on learning not just learning alone
>we learn with our whole body and we just don't
''MB Annotation'': trails off: "we just don't..." what?
!!!!!Code 14: 1st person
''Frequency'': 6
''Description'': personal example / experience / I / me / my
>I have a bio-kinetic memory
>what my fingers are doing
>in my case I think when we learn we learn with our whole body
>Hands-on learning for me
>Like sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember
''MB Annotation'': specific example from personal experience
>I think when we learn we learn with our whole body and we just don't
''MB Annotation'': 1st person plural ("we")
!!!!!Code 15: 2nd person
''Frequency'': 2
''Description'': abstract person / you / you're / your
>you're actually applying current uhh current information that you know of and you apply it to new information so you are accommodating and assimilating information but you are holding it in to your own understanding
>you're not not just learning about it technically speaking from a book but you're actually [...] engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body
!!!!!Code 16: specification: irregular
''Frequency'': 1
>Like sometimes I forget my password
''MB Annotation'': sometimes, infrequently
!!!!!Code 17: specification: definitive
''Frequency'': 5
''Description'': actually / just [only] / at least, technically speaking [exact meaning]
>Hands-on learning for me is when you're actually applying current uhh current information that you know of and you apply it to new information
''MB Annotation'': actually
>at least in my case I think when we learn we learn with our whole body
''MB Annotation'': at least in my case
>you're not not just learning about it technically speaking from a book
''MB Annotation'': not just ... technically speaking
>actually [...] engaged in the act of learning
''MB Annotation'': actually
>I have to pretend like I'm actually checking in at home
''MB Annotation'': actually
!!!!!Code 18: indigenous typology *
''Frequency'': 4
''Description'': jargon, words seldom seen/heard outside of the context of the conversation
>accommodating
''MB Annotation'': adapting, conforming to, transitioning
>assimilating
''MB Annotation'': learning, absorbing, taking in, amassing
>biodynamics, bio-kinesis
''MB Annotation'': referring to biological, physical processes
>bio-kinetic memory
''MB Annotation'': referring to a memory for biological/physical/bodily movement, a muscle memory
!!!!!Code 19: repetition *
''Frequency'': 4
''Description'': repeated words / phrases
>current uhh current
''MB Annotation'': current repeated (misspoken?)
>applying current uhh current information that you know of and you apply it to new information
''MB Annotation'': apply repeated - apply without a specified object of the application (former), and apply with an object (new information, latter)
>biodynamics, bio-kinesis, like you're whole body
''MB Annotation'': synonymous terms?
>not not
''MB Annotation'': likely (misspoken?)
!!!!!Code 20: transition *
''Frequency'': 2
''Description'': beginning a new thought / idea
>[elp [me guess] the password so I think we learn [...]
''MB Annotation'': "so I think" = start of summary
>Like sometimes
''MB Annotation'': start of example
!!!!!Code group: meta-level structure
''Frequency'': 3
''Description'': the high-level sequence of ideas in the excerpt
!!!!!Code 21: idea 1: transfer of knowledge
''Frequency'': 1
>when you're actually applying current uhh current information that you know of and you apply it to new information so you are accommodating and assimilating information but you are holding it in to your own understanding [...] or in a way that makes [...] but in a process
!!!!!Code: 22 idea 2: involvement of the body
''Frequency'': 2
>not just learning about it technically speaking from a book but you're actually [...] engaged in the act of learning about the path of uh... biodynamics, bio-kinesis, like you're whole body
>so I think we learn, at least in my case I think when we learn we learn with our whole body
!!!!!Code 23: example (1)
''Frequency'': 1
>Like sometimes I forget my password and I have to pretend like I'm actually checking in at home so that I also remember. So I have a bio-kinetic memory of what my fingers are doing in order to help [me guess] the password
''MB Annotation'': example also introduces new idea: importance of context in learning and remembering.
!!!!!Code 24: summary
''Frequency'': 1
''Description'': wrapping up
>so I think we learn, at least in my case I think when we learn we learn with our whole body
''MB Annotation'': reiteration of idea 2
!!!!Summary:
At the micro level of analysis, the structure of ideas conveyed by the speaker is beginning to appear, as reflected in the process detailed above. Preliminary codes were based on word frequency, co-occurrences, and collapsing words with similar meanings together. After the first round of locating and the instances of these codes and their surroundings, also known as key words in context*, it was possible to again collapse codes into higher-level semantic groups: cognitive activity, sequences, and specific objects. For instance, know, learn, and memory were semantically related as cognitive activity. Meanwhile, structure-based coding was also taking place, based on the placement of transitions, and a switching of tense (from 2nd to 1st). The combination of semantic code grouping and structural grouping suggests the meta-level codes detailed above, the presence of 2 main ideas in the text followed by an example: hands-on learning involves transferring information, it involves the body. The example hints at another idea: the importance of context in learning and remembering.
!!!!Attempt at restating the text, given the analysis detailed above:
Hands-on learning is a process by which existing knowledge is transferred and applied to a novel context. It is also a concrete physical process, involving bodily movement, complementing abstract processes of learning, such as through reading. Remembering a process in a new context is facilitated by hands-on learning, and can be decomposed into 2 steps: recalling the original context where the process was learned and replicating the process鳩cal movement.
''Note'': Codes and techniques as suggested by: G. W. Ryan and H. R. Bernard. Techniques to identify themes. In Field Methods (15)1:85-109, Sage 2003.
!!Part 2
//Discuss the pros and cons of using computer assisted data analysis when doing interpretive/critical research.//
!!!Pros
''Scalability'': given the processing power and available disk space of modern personal computers, large data sets are no longer a problem for most qualitative data analysis projects. This is especially true for large text data sets, such as interview transcripts or digitized field notes, however currently available data analysis software are also beginning to accommodate large corpuses of rich image, audio, and video data. (Of course, exploiting the scalability of computer-assisted qualitative data analysis often means purchasing the full non-trial version of applications).
''Data preprocessing'': without the aid of software, transforming the data into a consistent, approachable format can be extremely tedious. Speech recognition software, such as Dragon Naturally Speaking and Dictation, is improving to the point where interviews and dictations can be transformed reliably from audio to text, requiring little post-hoc manual corrections, sparing the analyst of hours of manual transcription. Audio processing tools, such as the free cross-platform Audacity application, also help to remove noise and distortion from audio files.鬡rly, handwriting recognition hardware and software are also improving, allowing for the transformation of handwritten field notes or documents to plain text. ~SmartPens and tablet computers are examples of such hardware devices. 崥rogenous data set can also be a headache, especially when most qualitative data analysis applications require a consistent file type and format, such as a plain text file. Document conversion applications and scripting languages can help to create a consistent data set, saving the analyst the hassle of manually extracting and/or converting data from each source file.
''Data formats'': Above, I mentioned the capabilities of analysis software with respect to text, image, audio, and video data. However, computer assisted data analysis software can acquire and support a range of other media. For instance, Texifter䩳coverText can be used to retrieve social media content for qualitative data analysis, namely Twitter and Facebook posts, as well as comments on blog posts and news sites. Gathering such data manually would be tedious and subject to various biases.
''Difficult / tedious calculations'': as mentioned by Ryan and Bernard (2003), some techniques to identify themes or codes in large text datasets can be accomplished quickly and with less errors if automated by software tools. These techniques range from straightforward word frequency counts and keyword-in-context extraction, to more sophisticated techniques, such as constructing a word co-occurrence matrix or applying a multidimensional scaling (MDS) algorithm to a large collection of text documents. The latter is a means to get a sense of the high-level thematic structure of the dataset by comparing mutual distances between individual documents based on the words they contain. These distance calculations are non-trivial, and MDS for large document sets can require an extensive amount of computational resources and time, from minutes to hours. Often, a human analyst, even one with a strong mathematical bent, operating at their maximum level of efficiency, could not perform all of these calculations within the course of their lifespan.
''Transparency'': analysis software, such as ~HyperRESEARCH, has source tracking features for easily tracing the analysts寲ies to their codes, annotations, and raw data. Filtering and search features are also helpful here. 鳠also possible to hyperlink to source material in theses and manuscripts submitted to academic publications. Given that academic publishing is now predominantly digital, there are often no size limits on appendices or supplemental material, which could contain linked source material. This model is beginning to appear in online journalism, such as http://www.propublica.org, where sources are embedded within hyperlinks in articles posted.
''Collaboration'': A distributed team of data analysts can collaboratively perform qualitative data analysis using a handful of computer tools. Digitized source files can be shared with cloud computing tools, such as ~DropBox, or with secure servers. Similarly, the products of analysis, such as theoretical writing or annotated/coded source data, can be worked on collaboratively. At this point, I am unaware as to the extent to which the various qualitative analytical software support collaboration of multiple analysts. Nevertheless, the technology exists: multiple users can edit and comment on text documents in Microsoft Office, or tag and comment photos on Flickr or Facebook.
*''Crowdsourcing your analysis'': the micro-work job market on the web is growing. Offering small monetary incentives for thematically tagging images, audio, videos, or text excerpts in your data set can be an effective way to analyze a large dataset and/or boost inter-rater reliability and validity. For example, Google has used this method for thematically tagging images returned by its search engine. Amazon壨anical Turk is such a service that provides an infrastructure for hosting and offering these micro-jobs to hundreds or thousands of individuals looking to make a few extra dollars.
*''Analytical provenance'': software with collaborative features solves a handful of problems, including ⥠did this source data come from?�added it to the dataset?�࣯ded this data?鴨out collaborative analysis software, tracking this information is often idiosyncratic and prone to confusion.
''Secure backups'': Your non-digitized source files might vanish in a fire or accident, they may even be stolen. Computerized raw data and/or products of analysis can also fall victim to such fates, not to mention corrupted hard drives, accidental deletions, and viruses. However, there exists ways to recover and protect your research materials. Iᬲeady mentioned the use of ~DropBox and secure servers. Version control applications, such as Subversion or Apple魥 Machine, are also important recovering documents, reverting to prior versions, and tracking changes over time.
!!!Cons
''Misconceptions'':
*''Scalability'': just because your software can handle large amounts of data doesn壥ssitate large amounts of data. Keeping your dataset within a reasonable size, a size that is appropriate given your time constraints and analysis goals, is more important than exploring the bounds of the software.
*''Roles of researcher and analytical software'': the analysis software is a tool like any other. It will not replace the role of the researcher. The researcher should not expect analytical software to do the data analysis for them, to expect that at the push of a button, the software will ingest raw data, code it, and display comprehensive results.
*''Time investment'': qualitative data analysis software may speed up certain parts of the analysis pipeline, such as data pre-processing and the communication overhead of collaborative analysis. Despite this, the time spent doing actual data-level analysis, scrutinizing the raw data, will not be any different than conducting analysis by hand. In fact, when one is beginning to use data analysis software, there may be a significant learning curve, and in such cases it may be faster to conduct data analysis by hand. However, with additional practice and gained familiarity with the software, this time gap will close.
''Techniques to identify themes / codes too objective'': Many of the techniques suggested by Ryan and Bernard (2003) are highly quantitative, especially those based on word frequency, repetition, and co-occurrence. Techniques such as MDS ignore word order, instead turning a document into a mathematical vector of words; the thematic structure identified by this technique reflects the presence and absence of words in a document, not their order or emphasis (or lack thereof). A critically important theme or code may only appear once in a source, or appear in a specific context or word ordering. It may not even appear at all: missing data can be difficult to detect using these techniques.
''Unsupported raw data formats'': not all raw data can be ingested into analysis software. Distorted handwriting or bad photocopying can make text extraction difficult, whereas a human observer may be able to easily read such documents. Similarly, audio recordings in noisy environments prove to be a problem for speech recognition tools, despite the existence of robust noise and distortion removal algorithms.
''Cost'': data preprocessing and analysis software packages can be expensive, and it may be difficult to justify the cost to supervisors and departments. Free trial versions are common for most software packages, however these are often limited in terms of dataset size and exporting/saving functions. The researcher should assess how much analysis of qualitative data will be required during the course of their project(s), how much time they would save by using analysis tools, and how much their time is worth.
''Ergonomics and eye strain'': as mentioned above, computer-assisted data analysis still requires a lot time spent scrutinizing the data, meaning long hours sitting in front of a computer. Despite ergonomic office furniture and LCD monitors, long periods of sitting, fatigue, and eye strain are workplace hazards that can be easily avoided. There may be parts of the data analysis pipeline that do not strictly require the use of a computer. Interleaving computerized and by-hand analysis methods may spare the researcher work-related injury. An added benefit of multiple analysis modalities is the possibility for triangulation of methods. For instance, comparing the results of thematic coding as done using software to that of physical cutting and sorting data excepts, perhaps colour-coding and taping them to a wall, may lead to new insights.
''Source'': G. W. Ryan and H. R. Bernard. Techniques to identify themes. In Field Methods (15)1:85-109, Sage 2003
/***
|''Name:''|EasyEditPlugin|
|''Description:''|Lite and extensible Wysiwyg editor for TiddlyWiki.|
|''Version:''|1.3.3|
|''Date:''|Dec 21,2007|
|''Source:''|http://visualtw.ouvaton.org/VisualTW.html|
|''Author:''|Pascal Collin|
|''License:''|[[BSD open source license|License]]|
|''~CoreVersion:''|2.1.0|
|''Browser:''|Firefox 2.0; InternetExplorer 6.0|
!Demo
*On the plugin [[homepage|http://visualtw.ouvaton.org/VisualTW.html]], see [[WysiwygDemo]] and use the {{{write}}} button.
!Installation
#import the plugin,
#save and reload,
#use the <<toolbar easyEdit>> button in the tiddler's toolbar (in default ViewTemplate) or add {{{easyEdit}}} command in your own toolbar.
! Useful Addons
*[[HTMLFormattingPlugin|http://www.tiddlytools.com/#HTMLFormattingPlugin]] to embed wiki syntax in html tiddlers.<<br>>//__Tips__ : When this plugin is installed, you can use anchor syntax to link tiddlers in wysiwyg mode (example : #example). Anchors are converted back and from wiki syntax when editing.//
*[[TaggedTemplateTweak|http://www.TiddlyTools.com/#TaggedTemplateTweak]] to use alternative ViewTemplate/EditTemplate for tiddler's tagged with specific tag values.
!Configuration
|Buttons in the toolbar (empty = all).<<br>>//Example : bold,underline,separator,forecolor//<<br>>The buttons will appear in this order.| <<option txtEasyEditorButtons>>|
|EasyEditor default height | <<option txtEasyEditorHeight>>|
|Stylesheet applied to the edited richtext |[[EasyEditDocStyleSheet]]|
|Template called by the {{{write}}} button |[[EasyEditTemplate]]|
!How to extend EasyEditor
*To add your own buttons, add some code like the following in a systemConfig tagged tiddler (//use the prompt attribute only if there is a parameter//) :
**{{{EditorToolbar.buttons.heading = {label:"H", toolTip : "Set heading level", prompt: "Enter heading level"};}}}
**{{{EditorToolbar.buttonsList +=",heading";}}}
*To get the list of all possible commands, see the documentation of the [[Gecko built-in rich text editor|http://developer.mozilla.org/en/docs/Midas]] or the [[IE command identifiers|http://msdn2.microsoft.com/en-us/library/ms533049.aspx]].
*To go further in customization, see [[Link button|EasyEditPlugin-LinkButton]] as an example.
!Code
***/
//{{{
var geckoEditor={};
var IEeditor={};
config.options.txtEasyEditorHeight = config.options.txtEasyEditorHeight ? config.options.txtEasyEditorHeight : "500px";
config.options.txtEasyEditorButtons = config.options.txtEasyEditorButtons ? config.options.txtEasyEditorButtons : "";
// TW2.1.x compatibility
config.browser.isGecko = config.browser.isGecko ? config.browser.isGecko : (config.userAgent.indexOf("gecko") != -1);
config.macros.annotations = config.macros.annotations ? config.macros.annotations : {handler : function() {}}
// EASYEDITOR MACRO
config.macros.easyEdit = {
handler : function(place,macroName,params,wikifier,paramString,tiddler) {
var field = params[0];
var height = params[1] ? params[1] : config.options.txtEasyEditorHeight;
var editor = field ? new easyEditor(tiddler,field,place,height) : null;
},
gather: function(element){
var iframes = element.getElementsByTagName("iframe");
if (iframes.length!=1) return null
var text = "<html>"+iframes[0].contentWindow.document.body.innerHTML+"</html>";
text = config.browser.isGecko ? geckoEditor.postProcessor(text) : (config.browser.isIE ? IEeditor.postProcessor(text) : text);
return text;
}
}
// EASYEDITOR CLASS
function easyEditor(tiddler,field,place,height) {
this.tiddler = tiddler;
this.field = field;
this.browser = config.browser.isGecko ? geckoEditor : (config.browser.isIE ? IEeditor : null);
this.wrapper = createTiddlyElement(place,"div",null,"easyEditor");
this.wrapper.setAttribute("easyEdit",this.field);
this.iframe = createTiddlyElement(null,"iframe");
this.browser.setupFrame(this.iframe,height,contextualCallback(this,this.onload));
this.wrapper.appendChild(this.iframe);
}
easyEditor.prototype.onload = function(){
this.editor = this.iframe.contentWindow;
this.doc = this.editor.document;
if (!this.browser.isDocReady(this.doc)) return null;
if (!this.tiddler.isReadOnly() && this.doc.designMode.toLowerCase()!="on") {
this.doc.designMode = "on";
if (this.browser.reloadOnDesignMode) return false; // IE fire readystatechange after designMode change
}
var internalCSS = store.getTiddlerText("EasyEditDocStyleSheet");
setStylesheet(internalCSS,"EasyEditDocStyleSheet",this.doc);
this.browser.initContent(this.doc,store.getValue(this.tiddler,this.field));
var barElement=createTiddlyElement(null,"div",null,"easyEditorToolBar");
this.wrapper.insertBefore(barElement,this.wrapper.firstChild);
this.toolbar = new EditorToolbar(this.doc,barElement,this.editor);
this.browser.plugEvents(this.doc,contextualCallback(this,this.scheduleButtonsRefresh));
this.editor.focus();
}
easyEditor.SimplePreProcessoror = function(text) {
var re = /^<html>(.*)<\/html>$/m;
var htmlValue = re.exec(text);
var value = (htmlValue && (htmlValue.length>0)) ? htmlValue[1] : text;
return value;
}
easyEditor.prototype.scheduleButtonsRefresh=function() { //doesn't refresh buttons state when rough typing
if (this.nextUpdate) window.clearTimeout(this.nextUpdate);
this.nextUpdate = window.setTimeout(contextualCallback(this.toolbar,EditorToolbar.onUpdateButton),easyEditor.buttonDelay);
}
easyEditor.buttonDelay = 200;
// TOOLBAR CLASS
function EditorToolbar(target,parent,window){
this.target = target;
this.window=window;
this.elements={};
var row = createTiddlyElement(createTiddlyElement(createTiddlyElement(parent,"table"),"tbody"),"tr");
var buttons = (config.options.txtEasyEditorButtons ? config.options.txtEasyEditorButtons : EditorToolbar.buttonsList).split(",");
for(var cpt = 0; cpt < buttons.length; cpt++){
var b = buttons[cpt];
var button = EditorToolbar.buttons[b];
if (button) {
if (button.separator)
createTiddlyElement(row,"td",null,"separator").innerHTML+=" ";
else {
var cell=createTiddlyElement(row,"td",null,b+"Button");
if (button.onCreate) button.onCreate.call(this, cell, b);
else EditorToolbar.createButton.call(this, cell, b);
}
}
}
}
EditorToolbar.createButton = function(place,name){
this.elements[name] = createTiddlyButton(place,EditorToolbar.buttons[name].label,EditorToolbar.buttons[name].toolTip,contextualCallback(this,EditorToolbar.onCommand(name)),"button");
}
EditorToolbar.onCommand = function(name){
var button = EditorToolbar.buttons[name];
return function(){
var parameter = false;
if (button.prompt) {
var parameter = this.target.queryCommandValue(name);
parameter = prompt(button.prompt,parameter);
}
if (parameter != null) {
this.target.execCommand(name, false, parameter);
EditorToolbar.onUpdateButton.call(this);
}
return false;
}
}
EditorToolbar.getCommandState = function(target,name){
try {return target.queryCommandState(name)}
catch(e){return false}
}
EditorToolbar.onRefreshButton = function (name){
if (EditorToolbar.getCommandState(this.target,name)) addClass(this.elements[name].parentNode,"buttonON");
else removeClass(this.elements[name].parentNode,"buttonON");
this.window.focus();
}
EditorToolbar.onUpdateButton = function(){
for (b in this.elements)
if (EditorToolbar.buttons[b].onRefresh) EditorToolbar.buttons[b].onRefresh.call(this,b);
else EditorToolbar.onRefreshButton.call(this,b);
}
EditorToolbar.buttons = {
separator : {separator : true},
bold : {label:"B", toolTip : "Bold"},
italic : {label:"I", toolTip : "Italic"},
underline : {label:"U", toolTip : "Underline"},
strikethrough : {label:"S", toolTip : "Strikethrough"},
insertunorderedlist : {label:"\u25CF", toolTip : "Unordered list"},
insertorderedlist : {label:"1.", toolTip : "Ordered list"},
justifyleft : {label:"[\u2261", toolTip : "Align left"},
justifyright : {label:"\u2261]", toolTip : "Align right"},
justifycenter : {label:"\u2261", toolTip : "Align center"},
justifyfull : {label:"[\u2261]", toolTip : "Justify"},
removeformat : {label:"\u00F8", toolTip : "Remove format"},
fontsize : {label:"\u00B1", toolTip : "Set font size", prompt: "Enter font size"},
forecolor : {label:"C", toolTip : "Set font color", prompt: "Enter font color"},
fontname : {label:"F", toolTip : "Set font name", prompt: "Enter font name"},
heading : {label:"H", toolTip : "Set heading level", prompt: "Enter heading level (example : h1, h2, ...)"},
indent : {label:"\u2192[", toolTip : "Indent paragraph"},
outdent : {label:"[\u2190", toolTip : "Outdent paragraph"},
inserthorizontalrule : {label:"\u2014", toolTip : "Insert an horizontal rule"},
insertimage : {label:"\u263C", toolTip : "Insert image", prompt: "Enter image url"}
}
EditorToolbar.buttonsList = "bold,italic,underline,strikethrough,separator,increasefontsize,decreasefontsize,fontsize,forecolor,fontname,separator,removeformat,separator,insertparagraph,insertunorderedlist,insertorderedlist,separator,justifyleft,justifyright,justifycenter,justifyfull,indent,outdent,separator,heading,separator,inserthorizontalrule,insertimage";
if (config.browser.isGecko) {
EditorToolbar.buttons.increasefontsize = {onCreate : EditorToolbar.createButton, label:"A", toolTip : "Increase font size"};
EditorToolbar.buttons.decreasefontsize = {onCreate : EditorToolbar.createButton, label:"A", toolTip : "Decrease font size"};
EditorToolbar.buttons.insertparagraph = {label:"P", toolTip : "Format as paragraph"};
}
// GECKO (FIREFOX, ...) BROWSER SPECIFIC METHODS
geckoEditor.setupFrame = function(iframe,height,callback) {
iframe.setAttribute("style","width: 100%; height:" + height);
iframe.addEventListener("load",callback,true);
}
geckoEditor.plugEvents = function(doc,onchange){
doc.addEventListener("keyup", onchange, true);
doc.addEventListener("keydown", onchange, true);
doc.addEventListener("click", onchange, true);
}
geckoEditor.postProcessor = function(text){return text};
geckoEditor.preProcessor = function(text){return easyEditor.SimplePreProcessoror(text)}
geckoEditor.isDocReady = function() {return true;}
geckoEditor.reloadOnDesignMode=false;
geckoEditor.initContent = function(doc,content){
if (content) doc.execCommand("insertHTML",false,geckoEditor.preProcessor(content));
}
// INTERNET EXPLORER BROWSER SPECIFIC METHODS
IEeditor.setupFrame = function(iframe,height,callback) {
iframe.width="99%"; //IE displays the iframe at the bottom if 100%. CSS layout problem ? I don't know. To be studied...
iframe.height=height.toString();
iframe.attachEvent("onreadystatechange",callback);
}
IEeditor.plugEvents = function(doc,onchange){
doc.attachEvent("onkeyup", onchange);
doc.attachEvent("onkeydown", onchange);
doc.attachEvent("onclick", onchange);
}
IEeditor.isDocReady = function(doc){
if (doc.readyState!="complete") return false;
if (!doc.body) return false;
return (doc && doc.getElementsByTagName && doc.getElementsByTagName("head") && doc.getElementsByTagName("head").length>0);
}
IEeditor.postProcessor = function(text){return text};
IEeditor.preProcessor = function(text){return easyEditor.SimplePreProcessoror(text)}
IEeditor.reloadOnDesignMode=true;
IEeditor.initContent = function(doc,content){
if (content) doc.body.innerHTML=IEeditor.preProcessor(content);
}
function contextualCallback(obj,func){
return function(){return func.call(obj)}
}
Story.prototype.previousGatherSaveEasyEdit = Story.prototype.previousGatherSaveEasyEdit ? Story.prototype.previousGatherSaveEasyEdit : Story.prototype.gatherSaveFields; // to avoid looping if this line is called several times
Story.prototype.gatherSaveFields = function(e,fields){
if(e && e.getAttribute) {
var f = e.getAttribute("easyEdit");
if(f){
var newVal = config.macros.easyEdit.gather(e);
if (newVal) fields[f] = newVal;
}
this.previousGatherSaveEasyEdit(e, fields);
}
}
config.commands.easyEdit={
text: "write",
tooltip: "Edit this tiddler in wysiwyg mode",
readOnlyText: "view",
readOnlyTooltip: "View the source of this tiddler",
handler : function(event,src,title) {
clearMessage();
var tiddlerElem = document.getElementById(story.idPrefix + title);
var fields = tiddlerElem.getAttribute("tiddlyFields");
story.displayTiddler(null,title,"EasyEditTemplate",false,null,fields);
return false;
}
}
config.shadowTiddlers.ViewTemplate = config.shadowTiddlers.ViewTemplate.replace(/\+editTiddler/,"+editTiddler easyEdit");
config.shadowTiddlers.EasyEditTemplate = config.shadowTiddlers.EditTemplate.replace(/macro='edit text'/,"macro='easyEdit text'");
config.shadowTiddlers.EasyEditToolBarStyleSheet = "/*{{{*/\n";
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar {font-size:0.8em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".editor iframe {border:1px solid #DDD}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar td{border:1px solid #888; padding:2px 1px 2px 1px; vertical-align:middle}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar td.separator{border:0}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .button{border:0;color:#444}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .buttonON{background-color:#EEE}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar {margin:0.25em 0}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .boldButton {font-weight:bold}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .italicButton .button {font-style:italic;padding-right:0.65em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .underlineButton .button {text-decoration:underline}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .strikeButton .button {text-decoration:line-through}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .unorderedListButton {margin-left:0.7em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .justifyleftButton .button {padding-left:0.1em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .justifyrightButton .button {padding-right:0.1em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .justifyfullButton .button, .easyEditorToolBar .indentButton .button, .easyEditorToolBar .outdentButton .button {padding-left:0.1em;padding-right:0.1em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .increasefontsizeButton .button {padding-left:0.15em;padding-right:0.15em; font-size:1.3em; line-height:0.75em}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .decreasefontsizeButton .button {padding-left:0.4em;padding-right:0.4em; font-size:0.8em;}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .forecolorButton .button {color:red;}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet += ".easyEditorToolBar .fontnameButton .button {font-family:serif}\n" ;
config.shadowTiddlers.EasyEditToolBarStyleSheet +="/*}}}*/";
store.addNotification("EasyEditToolBarStyleSheet", refreshStyles);
config.shadowTiddlers.EasyEditDocStyleSheet = "/*{{{*/\n \n/*}}}*/";
if (config.annotations) config.annotations.EasyEditDocStyleSheet = "This stylesheet is applied when editing a text with the wysiwyg easyEditor";
//}}}
/***
!Link button add-on
***/
//{{{
EditorToolbar.createLinkButton = function(place,name) {
this.elements[name] = createTiddlyButton(place,EditorToolbar.buttons[name].label,EditorToolbar.buttons[name].toolTip,contextualCallback(this,EditorToolbar.onInputLink()),"button");
}
EditorToolbar.onInputLink = function() {
return function(){
var browser = config.browser.isGecko ? geckoEditor : (config.browser.isIE ? IEeditor : null);
var value = browser ? browser.getLink(this.target) : "";
value = prompt(EditorToolbar.buttons["createlink"].prompt,value);
if (value) browser.doLink(this.target,value);
else if (value=="") this.target.execCommand("unlink", false, value);
EditorToolbar.onUpdateButton.call(this);
return false;
}
}
EditorToolbar.buttonsList += ",separator,createlink";
EditorToolbar.buttons.createlink = {onCreate : EditorToolbar.createLinkButton, label:"L", toolTip : "Set link", prompt: "Enter link url"};
geckoEditor.getLink=function(doc){
var range=doc.defaultView.getSelection().getRangeAt(0);
var container = range.commonAncestorContainer;
var node = (container.nodeType==3) ? container.parentNode : range.startContainer.childNodes[range.startOffset];
if (node && node.tagName=="A") {
var r=doc.createRange();
r.selectNode(node);
doc.defaultView.getSelection().addRange(r);
return (node.getAttribute("tiddler") ? "#"+node.getAttribute("tiddler") : node.href);
}
else return (container.nodeType==3 ? "#"+container.textContent.substr(range.startOffset, range.endOffset-range.startOffset).replace(/ $/,"") : "");
}
geckoEditor.doLink=function(doc,link){ // store tiddler in a temporary attribute to avoid url encoding of tiddler's name
var pin = "href"+Math.random().toString().substr(3);
doc.execCommand("createlink", false, pin);
var isTiddler=(link.charAt(0)=="#");
var node = doc.defaultView.getSelection().getRangeAt(0).commonAncestorContainer;
var links= (node.nodeType!=3) ? node.getElementsByTagName("a") : [node.parentNode];
for (var cpt=0;cpt<links.length;cpt++)
if (links[cpt].href==pin){
links[cpt].href=isTiddler ? "javascript:;" : link;
links[cpt].setAttribute("tiddler",isTiddler ? link.substr(1) : "");
}
}
geckoEditor.beforeLinkPostProcessor = geckoEditor.beforelinkPostProcessor ? geckoEditor.beforelinkPostProcessor : geckoEditor.postProcessor;
geckoEditor.postProcessor = function(text){
return geckoEditor.beforeLinkPostProcessor(text).replace(/<a tiddler="([^"]*)" href="javascript:;">(.*?)(?:<\/a>)/gi,"[[$2|$1]]").replace(/<a tiddler="" href="/gi,'<a href="');
}
geckoEditor.beforeLinkPreProcessor = geckoEditor.beforeLinkPreProcessor ? geckoEditor.beforeLinkPreProcessor : geckoEditor.preProcessor
geckoEditor.preProcessor = function(text){
return geckoEditor.beforeLinkPreProcessor(text).replace(/\[\[([^|\]]*)\|([^\]]*)]]/g,'<a tiddler="$2" href="javascript:;">$1</a>');
}
IEeditor.getLink=function(doc){
var node=doc.selection.createRange().parentElement();
if (node.tagName=="A") return node.href;
else return (doc.selection.type=="Text"? "#"+doc.selection.createRange().text.replace(/ $/,"") :"");
}
IEeditor.doLink=function(doc,link){
doc.execCommand("createlink", false, link);
}
IEeditor.beforeLinkPreProcessor = IEeditor.beforeLinkPreProcessor ? IEeditor.beforeLinkPreProcessor : IEeditor.preProcessor
IEeditor.preProcessor = function(text){
return IEeditor.beforeLinkPreProcessor(text).replace(/\[\[([^|\]]*)\|([^\]]*)]]/g,'<a ref="#$2">$1</a>');
}
IEeditor.beforeLinkPostProcessor = IEeditor.beforelinkPostProcessor ? IEeditor.beforelinkPostProcessor : IEeditor.postProcessor;
IEeditor.postProcessor = function(text){
return IEeditor.beforeLinkPostProcessor(text).replace(/<a href="#([^>]*)">([^<]*)<\/a>/gi,"[[$2|$1]]");
}
IEeditor.beforeLinkInitContent = IEeditor.beforeLinkInitContent ? IEeditor.beforeLinkInitContent : IEeditor.initContent;
IEeditor.initContent = function(doc,content){
IEeditor.beforeLinkInitContent(doc,content);
var links=doc.body.getElementsByTagName("A");
for (var cpt=0; cpt<links.length; cpt++) {
links[cpt].href=links[cpt].ref; //to avoid IE conversion of relative URLs to absolute
links[cpt].removeAttribute("ref");
}
}
config.shadowTiddlers.EasyEditToolBarStyleSheet += "\n/*{{{*/\n.easyEditorToolBar .createlinkButton .button {color:blue;text-decoration:underline;}\n/*}}}*/";
config.shadowTiddlers.EasyEditDocStyleSheet += "\n/*{{{*/\na {color:#0044BB;font-weight:bold}\n/*}}}*/";
//}}}
<<cite Crotty1998 bibliography:Bibliography>> ch.1
One's ''epistemology'' (theory of knowledge, how we know what we know) subsumes a ''theoretical perspective'', which in turn justifies the choice of ''research methodologies'', a strategy of research recipe in a sense, wherein individual ingredients are the ''methods'' themselves.
*Example: [[Constructionism|Epistemology: Constructionism]] (epistemology) > ''symbolic interactionism'' (theoretical perspective) > ''ethnography'' (methodology) > ''participant observation'' (method)
*Example: [[Objectivism|Epistemology: Objectivism]] (epistemology) > ''post-positivism'' (theoretical perspective) > ''survey research'' (methodology) > ''statistical analysis'' (method)
A note on ''ontology'', the study of being, the study of //what is//, whereas ''epistemology'' is the study of //what it means to know//. The existence of a world (ontology) without a mind is conceivable, but meaning without a mind is not (constructionist epistemology). Example: ''realism'' as an ontology and ''constructionism'' as an epistemology tend to be compatible.
//Note//: the distinction between ''quantitative'' and ''qualitative'' occurs at the level of methods, and is not as important a distinction as the distinction between theoretical perspectives or epistemologies. Some qualitative research is definitively positivist in nature. Likewise, quantification is not ruled out in interpretive or critical research. Can it be both objectivist and constructionist? While it may be contradictory, this is fine if you espouse a post-modernist theoretical perspective, a notion of ''fuzzy logic''. However many researchers are not comfortable with this stance, so we tend to be consistently objectivist or constructionist or subjectivist.
!Epistemology
*[[Objectivism|Epistemology: Objectivism]]: meaningful reality exists apart from consciousness; meaning is discovered.
*[[Constructionism|Epistemology: Constructionism]]: meaning comes into existence through an interaction between an object and a subject, meaning is constructed.
*''Subjectivism'': meaning is imposed on the object by the subject, with the object playing no part in the creation of meaning. Meaning is imported from elsewhere, from dreams, primordial archetypes, the collective unconscious, religious beliefs. Subsumes ''structuralism'', ''post-structuralism'', ''post-modernism''.
!!On knowledge & its justification (in class exercise):
Consider these claims. Justify or refute, provide basis for doing so.
*Nicotine-replacing gums and patches fail to help smokers stay off cigarettes
>@@color:#0000FF;While many smokers may place blame on the gums or patches for maintaining a smoking habit, this statement ignores varying levels of motivation. Wanting to smoke and wanting to quit are not polar extremes, but are somewhere along a continuum of motivation. Second, even if there was no clinical evidence for these products' effectiveness, I would expect that many users would be naive to the placebo effect, accomplishing the goal of quitting even if the products themselves didn't work.@@
*Adolescents who use drugs have dysfunctional family relationships
>@@color:#0000FF;Dysfunctional families attract attention. As a society we are fascinated by dysfunction, and it is reflected in our media and entertainment. Thus we have the case where we believe a correlation of drug-abusing adolescents with dysfunctional families to indicate causation in one direction or another (drugs lead to dysfunction or vice versa).@@
*Giving children grades in school will motivate them to do well
>@@color:#0000FF;Children may be motivated by role models or external influences.@@
!!My Own Epistemology
@@color:#0000FF;Based on a [[questionnaire|http://www.california.com/~eameece/questionnaire.htm]] given in class, I am apparently an ''essentialist'': //''universal ideas'' and ''divine archetypes'' are the source of knowledge//. I am somewhat ''intellectual'' (relying on reason and distrust feelings and experiences), and somewhat of a ''spiritualist'' (relying on spiritual methods to explain things).@@
Copied from [[Pulse Documentation|https://my.pulseenergy.com/help/reference/quantities]]
!!Apparent Power
Apparent power is the maximum real power that can be delivered to a load. It is the combination of true power (useful power) and Reactive Power. It is computed by multiplying the root-mean-square current by the root-mean-square voltage.
*''Note'': Apparent Power is typically used in the sizing of equipment or wiring.
Apparent Temperature
The perceived outdoor temperature caused by the combined effects of air temperature, relative humidity, and wind speed. The apparent temperature is a more complete measure of the total heat in the environment than air temperature alone.
!!Average Power
Average Power measures the average energy use over a window of time. It is a measure of the rate of energy use. Average Power will usually produce a smoother set of measurements than Instantaneous Power.
*''Tip'': Average Power is typically the most useful measure of power.
!!Cloud Cover
*Refers to the fraction of the sky obscured by clouds. It will be expressed as a percentage.
!!Counter
Some meters have a counter: an accumulating measurement that has no scale or unit. This kind of meter reading will need to be converted into a more useful unit of measurement. For example, it could be determined that every 4 ticks up could be equal to 1 kWh.
!!Current
Current measures the flow of electrical charge.
!!Dew Point
The temperature at which condensation occurs.
!!Downward Terrestrial Radiation
The downwelling component of longwave radiation. Its value at the surface is a measure of the greenhouse effect of the atmosphere. Terrestrial radiation is the longwave IR radiation that the earth emits because it is a warm body. Some of that is reflected by the atmosphere and comes back down again.
!!Energy
Energy measures the amount of work that can be performed by a force. The base unit of energy is the joule, though it can also be expressed in other units of measurement.
!!Flow
Flow measures the speed of a liquid in movement, for example: liters per minute or liters per hour.
!!Instantaneous Power
Instantaneous power is the power at an instant. It is equal to the instantaneous voltage times the current times the power factor.
*''Tip'': Instantaneous power is continuously varying with time and will show all spikes and anomalies. This is why Average Power is typically a more useful measurement.
!!Mass
A Mass point measures the mass of something. In Pulse, this will typically be steam.
!!Mass Flow
Mass flow is the movement of a mass per unit of time. For example, with steam it is usually measured in pounds per hour.
!!Net Radiation
The difference between absorbed and emitted radiation.
!!Number
Building occupancy information: number of occupants or rooms.
!!Percentage of Total Capacity
Percentage of Total Capacity is a measure of energy use in relation to its capacity at the measured location. Its unit of measurement is percentage.
!!Power Factor
Power factor is a measure of how effectively your equipment converts electric current from your utility company to useful power output, such as heat, light, or mechanical motion. The Power Factor measures what percentage of power is actually being used (Real Power/Apparent Power). A power quality meter can measure Power Factor directly. Otherwise, a calculation must done with the data from a typical hydro meter that only measures kWh and kVAR. This can be done automatically in Pulse using a Conversion Point.
*''Tip'': Faulty or oversized motors and generators can lead to a low Power Factor. A Power Factor below 90% or 0.9 can be subject to extra Power Factor charges from your utility provider to compensate for the reduced capacity and efficiency. There are measures you can take to correct a low Power Factor, for example: you can install Power Factor correction capacitors, controlling capacitors, and harmonic filters where appropriate.
!!Power Intensity
Power Intensity measures average power consumed, divided by the floor area of the building. The unit is watt/m2.
!!Pressure
The force per unit area exerted by the weight of the atmosphere above a point on or above the earth's surface.
!!Rate of Precipitation
The rate that any water-based particle falling from the sky (rain, snow, hail, etc.) reaches the ground.
!!Reactive Energy
A Reactive Energy point measures reactive energy, which is largely considered unusable and not capable of doing work. It is a calculable part of any electrical load. Reactive Energy is produced by the inductive motors in such items as: air conditioners, refrigerators, compact fluorescent light bulbs, and electric dryers.
!!Reactive Power
Reactive Power (kVar) is the electrical power strictly used to establish the electromagnetic field in transformers, lines, and motors. The ratio of your usable power consumption to your Reactive Power consumption determines your Power Factor.
!!Relative Humidity
Relative humidity is the ratio of the actual moisture content of the air to the potential moisture content. The ratio is often converted to a percentage.
!!Snow Cover Depth
The depth of snow on the ground.
!!State
The state of an energy consuming device such as a light switch or air conditioning, typically ON or OFF. In the system, OFF = 0 and ON = 1.
*''Tip'': In some circumstances it makes sense to transform a State Point into an Average Power Point when the Average Power is a constant value. For example, if you know that the bathroom lights draw exactly 500 kW when they are on, you could create a Conversion Point from State to Average Power where the Scaling Factor is 500 kW. Then when the lights are off, the Conversion Point reading will be 0 kW. When the lights are on, the reading will always be 500 kW.
*There are also circumstances where it useful to create a Conversion Point going from a demand Point to a State Point, such as when you want to build a model of when a device is on or off based on the readings only. For example, if the readings for a device are above 100kW it is "ON", if they are below 100 kW it is "OFF".
!!Surface Air Pressure
The pressure of the atmosphere at a given altitude or location.
!!Temperature
The degree of hotness or coldness of an environment.
!!Total Harmonic Distortion
The total harmonic distortion of a signal is defined as the ratio of the sum of the powers of all harmonic components to the power of the Fundamental frequency.
*''Note'': Total harmonic distortion is a measure of power quality. The lower the number the better.
!!Voltage
Voltage is the rate at which energy is drawn from a source that produces a flow of electricity in a circuit. It is a measure of electric potential.
!!Volume
How much three-dimensional space an object occupies.
!!Wet Bulb Temperature
Wet bulb temperature is the temperature measured by a wet bulb thermometer (which has a wet cloth sleeve that covers its bulb). It is the temperature you feel when your skin is wet and is exposed to moving air.
!!Wind Direction
The direction from which the wind is blowing.
*''Note'': More specifically it is the hourly average wind direction in degrees: 0ans North, 90ans East, 180ans South, and 270ans West.
!!Wind Speed
The rate of horizontal air movement past a given point
<<cite Crotty1998 bibliography:Bibliography>> ch.3
!Constructionism
An epistemology which posits that meaning is constructed rather than discovered, being both objective and subjective. Tied to the term ''Intentionality'', an interdependence between conscious subject and the object of the subject's consciousness. No true or valid interpretations, only useful and not useful.
Researchers as ''//bricoleurs//'', a jack-of-all-trades, making use of materials found on hand, rather than being self-reflexive or relying solely on conventional approaches and interpretations. Their task is to reinterpret.
The ''double hermeneutic'', the interpretation of the social world and its language, as opposed to the interpretation of the natural world and its language, which ignores the social world and the language it uses. The former must interpret an reinterpret to a greater degree.
''Constructionism'' places the social meaning (collective generation and transmission of meaning) at the centre of the interpretation, while ''constructivism'' involves the meaning-making of the individual during interaction with the object of their consciousness. The latter is not as critical as the former. Another distinction between the two is the less critical interpretation of the way things as they are (the sense we make of them), versus a critical analysis of the collectively shared meaning, the ''sedimentation'' of meaning built up over history.
!!Theoretical Perspective: Interpretivism
<<cite Crotty1998>> ch.4 - 5
Culturally derived, historically situated interpretations of the social world. A contrast between understanding and explaining (focus on causality) - the latter referring to ''Critical Inquiry''. Interpretivism is largely uncritical.
!!!Interpretivism: Symbolic Interactionism
''Methodologies'': [[Ethnography]], [[Grounded Theory]], [[Discourse Analysis]], [[Narrative Analysis]]
Interpretation of phenomena as it occurs within a cultural context, uncritical of culture. Putting one's self in the position of the other, relying on ''intersubjectivity'', ''language'', ''community'', ''communication''. The cultural understanding. Optimistic, progressivist, liberal, tolerant, pragmatic (//acquiescence in the social order//). Using language and other symbolic tools to understand the meaning of actors (those being studied).
The theoretical underpinning of ''Ethnography'' and ''Grounded Theory'' (meaning arising from the data, not from some other source), which takes place immersed within a culture. Related approaches: ''dramaturgical'', ''game theory'', ''negotiated-order theory'' (ever-shifting roles and organization of behaviour), ''labelling'' (study of deviants, in-group and out-group).
@@color:#0000FF;How can ''Symbolic Interactionism'' be the theoretical underpinning of ''Grounded Theory'', when GT implies that theory arises out of data and not from other sources (i.e. culture)? Wouldn't GT be more closely aligned w/ ''Phenomenology''?@@
!!!Interpretivism: Phenomenology
''//Back to the things themselves!//'' Interpretation of phenomena objectively, directly, and immediately, without explicit regard to culture, or treating culture with caution, suspicion. Explicitly uncritical of culture (although implicitly it is critical? as it doesn't acknowledge it). A fresh perspective, the unadulterated phenomena, an absence of acculturation (culture must be created anew). Belief that a new meaning will lead to a fuller meaning, that existing meanings are blindfolding us, that we take them for granted.
Related methods: unstructured interviews, studying experience from the point of view of the other, to identify, understand, describe, and maintain the subjective experience of the the other. A starting point for future interpretation.
When an objective focus is lost, inquiry becomes subjectivist and narcissistic. The world is not predetermined.
@@color:#0000FF;How critical is this line of thought? Is phenomenology tied to ''constructivism'', whereas ''symbolic interactionism'' is tied to ''constructionism''? The difference is still not clear.
An absence of cultural grounding equates to implicit cultural criticism?
Both ''Symbolic Interactionism'' and ''Phenomenology'' advocate putting //oneself in the position of the other//. Yet the latter is adamant that our observations remain unadulterated by culture. Are we not immersed in one culture or another? How can this be ignored when putting //oneself in the position of the other//?
''Phenomenology'': the lazy approach to interpretivism? Observing without prior understanding / preparation? Incomplete without subsequent cultural understanding and critique?@@
!!!Interpretivism: Hermeneutics
<<cite Crotty1998>> ch. 5
Interpretation (and sometimes critique) of text, art, and other artifacts. Origins in Biblical study. Deciphering indirect and hidden meanings, reflective. Different approaches involve more than the text alone, but also the intentions and histories of the creator, their historical context, the relationship between author and interpreter, the relevance for the reader. Going beyond the author's own understanding. Understanding a whole via its parts and the parts by //divining the whole//.
One approach (''Dilthey'') sees that historical context about universal spiritual forms, the author and the author's world is important for objective interpretation, while another approach (''Heidegger's Phenomenological Hermeneutics'') seeks to to //rid ourselves of our tendency to immediately interpret//, to focus on the ''phenomenology of being'', what he refers to as ''Dasein''. Meanwhile, the ''Gadamer's Historical Hermeneutics'' approach is to unify the historical interpretation, tradition, with the present's relevant interpretation, the reader's own interpretation, a unity of meaning. For this reason this approach cannot judge contemporary works as it their true nature in history is as of yet unknown. Combining our self-prejudices and the less-important individual judgments. In other words, the relevance to the current setting is important.
Intertwined with ''literary criticism'' and ''reading comprehension theory''. Debate over how privileged the reader should be - to what extent should the work's meaning be obvious? Is reading an activity of ''transmission'', of ''translation'', of ''interaction'', or of ''transaction'', a ''construction''? The latter is generative: the reader creates a personal interpretation and insight that goes beyond the author's intent, what is not written in the text. Reading as ''empathic'', seeing the world from the author's perspective, their perceptions, attitudes, and feelings.
''Demythologizing'' vs. ''demystification'': the former relates to how a text is reverenced and its hidden meaning is sought out. The latter posits that the text represents a false reality and a new interpretation is needed.
@@color:#0000FF;An understanding of journals, notes, personal correspondence - written material not intended for publication or wide readership, sometimes not intended to be read by anyone but the author for later reference. How to approach? Recall the <<cite Klahr1999>> paper which involves a re-interpretation of scientist's lab notes and journals during their process of scientific discovery.@@
!!Theoretical Perspective: Critical Inquiry
''Methodologies'': [[Action Research]], [[Critical Ethnography|Ethnography]], [[Dialectics]]
!!!Critical Inquiry: Marx, the Institute for Social Research
<<cite Crotty1998>> ch. 6
Critical inquiry addresses the "battleground of hegemonic interests", research that reads and understands interactions in a community in terms of conflict and oppression.
> //The philosophers have only interpreted the world in different ways. The point is to ''change'' it.// (Marx)
A theory that reflects the current situation vs. a theory that seeks to change the situation.
> //Life is not determined by consciousness, but consciousness by life... It is not the consciousness of men that determines their being, but, on the contrary, their social being determines their consciousness... All social life is essentially ''practical''.// (Marx).
Critical inquiry is motivated by the observation that language provides legitimacy, as long as oppressive forces do not have control over language, thought cannot be compromised and thus people can move freely with ideas.
The ''dialectic'' (owing to Hegel): succession of societal forms in history mirrors human self-understanding. Integral to Marx's view of history. Thesis and antithesis, their interaction leading to a synthesis. Later, Horkheimer sought after a dialectic that unified social science and positivist science that informs one another. The former values immediacy, the flux of direct experience with no time for empirical data. The latter reduces knowledge to what is statistically verifiable, robbing the experience of vitality.
Adorno's work addresses the existing philosophical notion of the conceptual, that while we need to conceptualize, we must be aware of the irreducibility of the non-conceptual. This is the tension between the conceptual and non-conceptual, of multiplying difference while preserving resemblance rather than assimilation through identification with the conceptual. Concepts should be tentative and suggestive rather than prescriptive. Recognizing the 'ungenuineness' of the genuine.
> //Objects do not go into their concepts without leaving a remainder.// (Adorno)
> //To perceive resemblances everywhere, making everything alike, is a sign of weak eyesight.// (Nietzsche)
> //We should seek to mimic what we experience as fully as possible rather than believing we can capture what we experience conceptually.// (Adorno)
The notion of ''mimesis'': reality should copy art rather than the other way around (Adorno).
Adamo's ''negative dialectics'', a critique of consciousness in turn informs a critique of society. A critique from within. To be critical of the conceptual and to balance with the conceptual with the individual is by no means a phenomenological reduction (''//Back to the things themselves!//'').
@@color:#0000FF;Do we justify the use of case studies with Adorno's position: that we must balance the conceptual with the individual? The genuine with the generalizable? Vitality and verifiability?
As a methodology, what methods do ''dialectics'' subsume or justify? Could this occur through the process of writing meta-reviews, survey papers, comparing and contrasting across a divided landscape of opinion and theory?@@
!!!Critical Inquiry: Habermas, Freire
<<cite Crotty1998>> ch. 7
''Mimesis'' is romantic irrationalism. Adorno abandoned immanent critique for total critique. Instead, Habermas addresses the sociopolitical reality of Western culture, that which reduces social relations to that of the objectified and commodifed, splitting object from subject, looking for control over nature. Habermas goes beyond Marx's focus on production to ground a socially and historically developing rationality, describing labour as instrumental action and social interaction as communicative action. Humans organize their experience in terms of cognitive interests.
Empirical sciences led by interest to predict and control. Historico-hermeneutical led by interest in achieving a practical, mutual, and intersubjective understanding. Finally the critical sciences with an intent to bring about emancipation from relations of dependence an ideology. The latter must be achieved through dialogue, through language. Learning must involve both moral-practical knowledge and empirical-analytic knowledge.
The ''ideal speech situation'': //unrestrained and universal discourse enabling an unconstrained consensus to emerge whereby the idea of truth can be analysed// (Habermas). Discourse assesses the validity of claims towards the better argument. Ethics as the normative basis of critical theory. //Reason. Language. Communication. Social Critique. A link between Marxism and radical democracy.// Economic AND intellectual emancipation. Where does ''false consciousness'' reside?
Freire was involved with literacy campaigns in Brazil beginning in the 1950s. Again language is the key to a critical consciousness, critical perception, critical thinking, or ''conscientisation'', a joint project between all humans engaged in dialogue. Humans, in solidarity with their world, must make use of their creative imagination to address human situations, to avoid being ''dehumanized'' through ''a culture of silence'', this being ''problemitization''. Reflection and action co-occurring, never in isolation, leading to further action and reflection on an ever-changing situation. Dialogue, the unity between subject and object, the objective and the subjective perception, fact and interpretation.
>//Consciousness is never a mere reflection of material reality but is a reflection upon material reality.//
>//Functionally, oppression is domesticating.//
Critical inquiry today, an ongoing project calls current ideology and power relations into question, initiates action, challenges commonly held values, assumptions. It is concerned with issues of power and oppression, of hegemony and injustice.
@@color:#0000FF;In HCI research, the dominant form being one of [[Action Research]], we are critical of existing practices and understandings involving the use of technology. We rely on ''case studies'' and contextual inquiries at formative stages, interviews, observations to break down the conceptual, to highlight differences and corner cases. But the dominant form of research at the ''summative'' stage is quantitative experimental research, the construction of new concepts and a detachment from the individual cases established in ''formative'' stages. We often praise a body of research if is highly replicable, generalizable. But what if our solutions are not generalizable or extendible to larger concepts? What if they include a critique of the non-conceptual, some individual behaviour. What if the subsequent improvement via some intervention is only suited to the situation identified at the formative stage.
What of critical research and [[Action Research]] that at its core does not address oppression and social injustice, but one that addresses those who satisfice, those who rely on inefficient, impractical, or cumbersome ways of interacting with objects and others. Is this, at its core, still a sociopolitical question? (If so, what is it?). Must all critical inquiry be sociopolitical in nature? Can we be oppressed by technology and the need of technology to find a use for itself, by our reliance on technology? Can we be oppressed by our own habits?
On the other hand, can technological intervention constitute action, permitting a critical consciousness. Does the advent of social networking facilitate dialogue, a dialectic?@@
<<cite Crotty1998 bibliography:Bibliography>> ch.2
!!Theoretical Perspective: Positivism
''Positivism'' is ''objectivist'' by definition. Not the opposite of negative, but in the sense that //something that is posited//. Key players: Auguste Comte, Francis Bacon. Meaning is discovered. We can only know what we observe. There exist true and false interpretations of meaning. Observation, experiment, and comparison.
!!Theoretical Perspective: Logical Positivism
The Vienna circle. The ''principle of verification''. No statement is meaningful unless if can be verified. Analytical statements (tautologies, contradictions) are easy to verify. Non-analytic, synthetic statements must be verified to become meaningful. This must occur by experience, by observation. The result is a meaningful fact. Metaphysics and theology produce statements than cannot be verified. An amount of ''reductionism'' is needed to verify statements from fields that build upon physics and philosophy.
Criticism: the scientific world is not the world we inhabit in our everyday lives. The social world, theology, metaphysics matter.
!!Theoretical Perspective: ~Post-Positivism
''Methodologies'': [[Experiments]] and [[Quasi-Experiments]], [[Regression discontinuity]], [[Survey Research]], [[Single Subject]]
Uncertainty is possible, probability (Heisenberg). Statements are relatively meaningful. ''Falsification'' rather than verification (Karl Popper): hypothesis testing. Every scientific statement must remain tentative until it is proven to be false.
Current criticism: On scientific revolutions (Kuhn), an anarchic (Feyerabend) tradition of challenging existing theory, modes of thinking. A shifting of ''paradigms''. The Popper stance of ''falsification'' is today's ruling paradigm, wherein //normal science is a complex and consuming mopping up operation//. Eventually, the current paradigm proves to be inadequate. At these junctures, a ''paradigm shift'' occurs, altering the way we view reality. This stance is loosening the grip of post-positivist thought on science. Feyerabend's view of modern science is one of post-positivist indoctrination, a threat to academic freedom. The proposed solution: ''counter-induction'', rather than proving a statement false, it //calls into question commonly-used concepts// by comparison, an external standard of criticism. For him, scientific beliefs are still a product of a socio-political and cultural reality, as are all beliefs, and are no longer truly objectivist. As a result, some thinkers adopt a a ''constructionist'' stance, while others humbly temper their objectivist stance, regarding findings as tentative.
@@color:#0000FF;I would suggest that qualitative HCI research is still objectivist in nature, that we claim validity and generalisability in our findings via saturation in data collection. Does constructionist research occur within HCI? A shift in formative design studies wherein the designer puts themselves in the place of the user? A critical inquiry of workplace practices or collaboration methods?@@
Fieldwork and writing about people. Participant observation is the major data collection method. In-situ informal interviewing is also a key data collection method. Impressionist and confessional styles of data representation.
Some other forms of ethnography:
*Critical ethnography - ethnography for social change, a focus on the exploited, alienated, disenfranchised (i.e. Pascoe's book)
**critical ethnography + intervention 䩯n research
*[[Autoethnography]] = writing about yourself w.r.t culture; subjectivity, reflective writing + theory, frameworks, cultural analysis (i.e. Pascoe's appendix)
**[[Autoethnography]] 䯢iography (narcissistic, more narrative than analysis),
**<<cite Lloyd2011 bibliography:Bibliography>> 䯭ethnography, but reflective writing for the purpose of becoming self-aware in design
!!Sources:
*[[Autoethnography|http://en.wikipedia.org/wiki/Autoethnography]] - Carolyn Ellis' auto-ethnographic writing
*[[Clifford Geertz|http://en.wikipedia.org/wiki/Clifford_Geertz]]'s impressionist ethnographic writing
*[[Ethnography - slides by Sandra Mathison|http://prezi.com/ftcr-ez_736n/ethnographies/]]
*[[Stefan Helmreich (MIT Anthropology)|http://web.mit.edu/anthropology/faculty_staff/helmreich/publications.html]]
*<<cite Lloyd2011 bibliography:Bibliography>> - auto-ethnography in design
<<cite Anderson2013>>: Ethnography is not only //what people do// but //why people think they do what they do//. It //transcends simplistic, goal-oriented analyses of action and belief//.
!![Lloyd2011]
<<cite Lloyd2011>> conducted a long-term case study with 3 domain specialists (crime and disorder reduction) over the course of 3 years. Over this time the authors took a human-centred approach to desigining a geovisualization. Mixed (qualitative and quantitative) evaluation and design methods were used, some of which were more effective than others. They report their findings and offer advice regarding best practices for geovisualization design, which largely generalize to other application domains and visualization types.
They advocate studying real domain experts in context. They suggest a master-apprentice relationship with occasionally swapped roles, rahter than a consultant/client relationship: they should be viewed as co-discoverers, colleagues, and partners, rather than subjects. The authors suggest using real data, understanding its context of use.
''Method'': They began with methods for understanding context of use: [[Contextual Inquiry]] with seveal content analysis techniques (word frequencies, keywords-in-context, networks of relationships, card sorting (cluster analysis)). This was supplemented with interviews and observation, along with the study of internal and external domain documents. This was followed by a series of methods for establishing requirements: the [[Volere Method]] (a template of structure questions), a lecture on possible/suggested geovis and ~InfoVis design techniques followed by card sorting, user sketching, and delayed recall of concepts. This stage also included scenario interviews using real domain data, and a questionnaire. Qualitative and quantitative data was derived from each of these methods. At this stage, the sketching and scenario sessions were more effective than the template questions, the questionnaire, or the card sorting. The next stage involved early prototypes (paper), where an [[Autoethnography]] approach was taken to the design process, a highly introspective and reflective method. Recorded interactive sessions with these prototypes involved the [[Think Aloud Protocol]]. The use of real domain data is critical at this stage for cognitive plausibility. Production values should be kept low, encouraging the idea that the design prototypes are transient - sketchiness should be afforded. They suggest hybrid means for generating low-fidelity prototypes ([[Patchwork Prototyping]]), using mixes of paper and ~InfoVis toolkits such as [[Processing]] or [[ProtoVis]]. Later paper and interactive prototypes were more interactive and retained the use of real domain data. They were evaluated in sessions mixing [[Chauffeured Prototyping]] and [[Wizard-of-Oz Prototyping]]. Again the [[Think Aloud Protocol]] was used. 7 sessions with 3 experts lasted 2 hours each, followed by an interview. They found that paper prototypes generated more suggestions than interactive prototypes (except for interface-related improvements).
Methodologically, this long 3-year process placed a strain on the relationship with domain experts. What was learned from these sessions was highly data-dependent. An emphasis on iteration was very important. Free exploration generated more suggestions than task-based directed interaction. Fig. 7 outlines which techniques worked best and which techniques should be avoided. The list on p. 2506 outlines their key recommendations:
#design process should be interactive, creative, interesting, involvement of all stakeholders
#use a range of real data known to the domain experts, use it early in the design process
#emphasize transience in the designs generated (use paper)
#scenarios with data are effective for design suggestions, and also for ~InfoVis education
#develop digital sketches that are flexible and re-usable, used in conjunction with paper and interactive prototypes
#free exploration with prototypes is encouraged
#use the [[Think Aloud Protocol]]
#assume an attitude of co-discovery - do not establish the consultant-client relationship
#use an [[Autoethnographic|Autoethnography]] approach to design
#iterate within and between levels of the HC process
!!!!Comments & Questions
*I hadn't previously been aware of the term [[Autoethnography]] - seems like a good practice for any form of design work
*An exhaustive account of 3 years of work - many methods attempted and a thorough trace of each methods back to their original sources
*Paper prototypes generated more suggestions than interactive prototypes (except for interface-related improvements) - what were the other suggestions relating to? Visual encoding? Visualization type? Task flow? (they later state, albeit vaguely, that paper was good for suggesting functionality and enhancements).
*They continually compare their methodology against that of the Human-Centered best practice outlined in ICO 13407 - which pertains to all interactive systems. By comparing against the HC approach, which is so general, they do not address the requirements specific to visualization systems: providing insight, enabling open-ended exploration and learning, supporting decision-making and problem-solving under uncertainty.
*MS recommended this paper from ~VisWeek - to appear in Vol. 17, Issue 12 of Transactions on Visualization and computer graphics
*Related work was sprinkled throughout, as opposed to being presented in a dedicated section. The organization of the paper follows their design stages.
*Some parts and figures not especially clear / concise.
*In the concluding section (and Fig. 8), they argue that some low-fidelity design (cheap, paper prototypes with real domain data) is required before initial grounded context-of-use study (as suggested by <<cite Isenberg2008>>). Researchers should alternatively begin with design and proceed to requirements or context-of-use phases, before iterating on design and proceeding to evaluation. There is not much discussion of iteration and lopping through this cycle in the paper - they admit that their design process was fairly linear, much like the sections of the paper.
*<<cite Crabtree2009>> - ethnography considered harmful (CHI '09)
!![Crabtree2009]
<<cite Crabtree2009>> acknowledge the increase of new ethnographic approaches reported in the CHI community, particularly for studying contexts outside of the workplace. These approaches have been appropriated by the community based on a misconception of what "work" is. The authors maintain that "work" pertains to actions and interactions with objects in //any// setting, be it the traditional //or// non-traditional workplace, the home, the museum, anywhere. The misconception is that "work" only pertains to workplace activities in workplace contexts, centred around productivity and efficiency, and not to other activities that are shaped by culture and social interaction. These new ethnographic approaches thereby focus too heavily on culture and context, on defamiliarizing activities and interactions with objects into literary abstractions, or on critique of the design process, choosing not to explicitly focus on the "work" being done, the situated actions and interactions. As a result, these methods are appropriate for the social sciences and typical deliverables of this field (being literary in nature), but not for informing design, as ethnographic approaches in the HCI community has historically been used for. Cultural abstractions, de-familiarized practices, design process critique, and exotic stories do not inform design, they are tourism for the HCI researcher.
The take-home message is that "work" is far more general than what happens in traditional workplaces, and that ethnographic research approaches should above all focus on situated action for the purpose of informing design.
!!!!Comments & Questions
*Appropriate to read for practitioners studying non-workplace contexts, or non-traditional workplaces, workplaces in other cultures; for those studying conventional workplaces, a focus on studying actions and interactions should have never shifted (so you're safe)
*I'd be curious to read about new ethnographic approaches studying traditional workplace settings, and their results.
!![Greenberg2008] - Usability Testing Considered Harmful
<<cite Greenberg2008 bibliography:Bibliography>>'s opinion article echoes a similar meta-review by <<cite Ellis2006>> regarding the (mis)use and purpose of evaluation in HCI research - in particular they write about the [[Usability Evaluation]], encompassing laboratory-based [[User Observation Studies|Laboratory Observation]], controlled [[User Studies|Controlled Experiments]], and [[Inspection Methods]].
>"Scientific evaluation methods do not necessarily imply science."^^1^^
They argue that [[Usability Evaluation]], applied at the wrong time in a design process, can kill promising design ideas prematurely, inhibiting creativity and innovation, resulting in a local-maxima design. In addition, the [[Usability Evaluation]] will not consider the contextual and cultural use of the design, or how it may be adopted to serve other purposes in the real world. The authors provide many examples of prior technological innovation (the radio, the automobile, Sutherland's Sktechpad, Englebart's NLS) that were not formally evaluated, and would have likely been stalled / discontinued had our current rigorous [[Usability Evaluation]] methodology been applied. Like <<cite Ellis2006>>, [[Usability Evaluation]] as it is conducted in many instances constitutes weak science, in that the results favour a particular system and hypothesis, an existence proof. The goals of the evaluation don't match the larger research goals. The scientific significance (i.e. discovery) and contribution trumps engineering and design innovation.
>"Do usability evaluations of toy deployments really test much of interest?"
The authors challenge the CHI community (researchers and reviewers), as well as educators and practitioners, to acknowledge and support other design and evaluation techniques, to place less importance on the need for [[Usability Evaluation]]. Where [[Usability Evaluation]] is used, replication is needed to confirm prior findings. A greater balance of subjective and objective evaluation methods are needed. They suggest looking to other fields (industrial design, architecture), where such a balance is common, particularly in early design stages. When [[Usability Evaluation]] is used at early stages, the methods focus on negative aspects of design, and positive aspects are lost as design alternatives are discarded.
The authors advocate design sketching prior to prototyping, going for a breadth of design alternatives rather than iterating on a single design. Evaluation should not treat sketches as prototypes, and should be in the form of design critiques (similar to those used in design and architecture fields), to establish both positive and negative aspects of design, acknowledging that usability problems have yet to be flushed out when the best design(s) are prototyped. Many great ideas (see list above) were not highly usable in the early design stage. The early design stage should be a time for questioning utility rather than usability, acknowledging the history of ideas that have found use outside of their initially envisioned use case (i.e. Bush's Memex to the the internet), despite usability problems. Relaxing the need for [[Usability Evaluation]] does not mean unjustified or unvalidated designs, but rather than they can and should be justified and validated in other ways, by truly understanding user requirements, case studies, examining cultural adoption and use, focus groups, design sketching, design critiques, etc. DIS, ~UbiComp, and [[CSCW]] are related fields who are already accustomed to a breadth of evaluation techniques.
!!!!Comments & Questions
*Discussed at MUX forum Nov. 2, 2011
*Read in CPSC 544, fall 2009
*^^1^^ <<cite Ellis2006>> argues that to carefully consider the purpose the evaluation will serve (i.e. whether, when, where, and with who will a technique work, rather than: will the technique work? or is this technique the best? the latter two can be answered by [[Usability Evaluation]], but don't constitute novel research (science)
**Evaluation should be regarded as explorative, rather than summative or formative, which are often not revealing or valuable when viewed from these perspectives. Often these evaluations are necessary for good product design, but they do not constitute good research, which is exploratory. Often authors of research papers include an evaluation even if the result is a foregone conclusion (i.e., even without a study, the best method is obvious) or if even when the evaluation result was not the goal of the research initiative. Other times, the evaluation would address something else, a technique or human behaviour, but not the system itself.
*the CHI community needs a review paper similar to <<cite Lam2011>>: guiding scenarios for evaluation
*curious how terms like //summative// and //formative// don't creep into the discussion
*KB sez:
>"It is somewhat surprising that the article by Greenberg and Buxton does not reference this article by Card and Newell (1985) [[http://dl.acm.org/citation.cfm?id=1453661.1453662]]" ... SY sez: "just as surprising is the fact that the ACM Bibliometrics notes a "citation count" of only 5 for the Card and Newell article."
*from Card and Newell (1985):
>"In any field, hard science [...] has a tendency to drive out softer sciences, even if the softer sciences have important contributions to make. It is possible that, as computer science and artificial intelligence contributions to human-computer interaction mature, this could happen to psychology. It is suggested that this trend might be prevented by hardening the applicable psychological science. [...] the resulting body of knowledge would be too low level, limited in scope, too late to affect computer technology, and too difficult to apply..."
!![Crabtree2009] - Ethnography Considered Harmful
<<cite Crabtree2009>> acknowledge the increase of new ethnographic approaches reported in the CHI community, particularly for studying contexts outside of the workplace. These approaches have been appropriated by the community based on a misconception of what "work" is. The authors maintain that "work" pertains to actions and interactions with objects in //any// setting, be it the traditional //or// non-traditional workplace, the home, the museum, anywhere. The misconception is that "work" only pertains to workplace activities in workplace contexts, centred around productivity and efficiency, and not to other activities that are shaped by culture and social interaction. These new ethnographic approaches thereby focus too heavily on culture and context, on defamiliarizing activities and interactions with objects into literary abstractions, or on critique of the design process, choosing not to explicitly focus on the "work" being done, the situated actions and interactions. As a result, these methods are appropriate for the social sciences and typical deliverables of this field (being literary in nature), but not for informing design, as ethnographic approaches in the HCI community has historically been used for. Cultural abstractions, de-familiarized practices, design process critique, and exotic stories do not inform design, they are tourism for the HCI researcher.
The take-home message is that "work" is far more general than what happens in traditional workplaces, and that ethnographic research approaches should above all focus on situated action for the purpose of informing design.
!!!!Comments & Questions
*Appropriate to read for practitioners studying non-workplace contexts, or non-traditional workplaces, workplaces in other cultures; for those studying conventional workplaces, a focus on studying actions and interactions should have never shifted (so you're safe)
*I'd be curious to read about new ethnographic approaches studying traditional workplace settings, and their results.
!![Kraut1988]
<<cite Kraut1988>>'s early CSCW paper examines scientific research collaboration by means of an interview study (N = 70), a survey study (N = 66), and an archival study of research publications and co-authorship pairings between researchers (N = 93). They find that patterns of communication and collaboration are directly affected by proximity of collaborating researchers, wherein low-cost, spontaneous, and high-quality multi-model communication is more likely to occur. These patterns transcend research interests and departmental ties: collaborations outside of one's field/department are more likely to occur if individuals are physically close.
The archival study examined 4 quantified four measures: collaboration (yes/no), organizational proximity, physical proximity, and research similarity. Their analysis centers around the quantitive relationships betwee these measures. Physical proximity leads to more collaboration, regardless or organizational sturcture or research similarity that divides individual researchers. They explain these results with other studies that examined communication frequency as a factor of distance, and the low-cost and high-quality communication physical proximity affords.
The paper concludes with design recommendations to improve research collaboration over distances, which is now dated (omnipresent high-quality 2-way video connection between research groups in a shared/communal space, encouraging spontaneous, high-quality and low-cost informal communication).
!!!!Comments & Questions
*the interview study involved 3 fields: computer science, social psychology, management science, whereas the survey study only involved psychologists. Finally the archival study considered 93 members of a single R&D organization. The latter study's results may not generalize beyond one company's work culture. Furthermore, it only considered researchers who had already co-authored at least two papers, which cannot account for researchers who have yet to collaborate, or collaborating researchers who do not generate an archived report. Most of this paper focuses on this dataset.
*I'm surprised they don't go into much detail of their rich qualitative data (interviews, surveys). The paper focuses on the quantitative evidence of their archival study. I was expecting a grounded analysis of the qualitative results.
*Methodological details regarding the analysis of their datasets are not given.
*Design implications are dated.
!![Nardi2000] - CSCW GT Eval
<<cite Nardi2000>> studied IM usage among 20 employees in various technical and managerial positions, distributed across a ~TelCo, an internet company, and independent contracting. They collected data from contextual interviews (some with audio and video), supplemented with IM logs over a period of time. They challenge several standing assumptions regarding informal communication from media theory which center around the notion that such communication is about information exchange: questions and clarifications, coordination and scheduling, efficient communication of facts, keeping in touch with friends and family. The researchers data reflected much more than this, that IM use supported //outeraction//: IM could be used to negotiate conversational availability, establish social connection and presence awareness, preserve conversational context across intermittent conversations, and manage communication medium choice/switching (i.e. using IM as a preamble to a phone conversation). Their findings overlap with past work employing ethnomedological and grounded conversational approaches, but contributed the additional understanding of outeraction.
!!!!Comments & Questions
*this paper's main focus is the discussion of the studies findings. The analysis details and methodological approach for conducting the study and collecting/analyzing data is not given.
!![Jennings2011] - Measuring Creativity
<<cite Jennings2011>> presents a quantitative research methodology for measuring exploratory creativity in an aesthetic visual search task. This task represents a broad set of artistic and creative visual tasks, including composition of a still-life photograph or painting.
The authors describe creativity as a search, and discern between //path search// (outcome is known, path uncertain) and //place search// (outcome is uncertain, means for achieving it are relatively straightforward). They argue that artists' outcomes depends on early decisions, seldom a matter of selecting between a few choices. Their intent is to study and make inferences about artists' search strategies as they explore a creative space (or search landscape) over time. This paper documents their experimental paradigm for capturing one's search trajectory (observable choices), allowing for a better understanding of one's search strategy (hidden to the observer, difficult to verbalize).
An artistic composition task involving the placement of a virtual camera and light source around a virtual scene is their task. They argue that this task captures more than insight or divergent thinking, however it is not as ecologically valid as an actual artistic composition task (painting / photographic a scene). However, their task allows for gains in precision, consistency, and accuracy.
Artistic domains tend to be //blind// in that transitions in a foggy search landscape are non systematic, often resulting in trial and error (see Campbell (1960): //Blind Variation and Selective Retention Theory//). Not all blind processes are random (however a purely random process is blind), nor are they brute force. There appears to exist a //metaheuristic// algorithm, a balance between //intensification// and //diversification// of search trajectories. Retention can also be blind: changes that make the solution worse are sometimes retained so as to move away from local maxima in a search space, despite not guaranteed to lead to higher maxima.
Each participant's search trajectory can be monitored, and their search landscape (ruggedness, fogginess) estimated based on their ratings for target images and their predicted ratings for adjacent (unseen) images. Together these can be used to draw inferences regarding one's search strategy. Their theoretical model is described as: Given a problem (an interface, an open-ended goal, and a scene), an observed search trajectory and solution, and by probing one's goal criteria, we can partially measure the landscape topology. With this information, search strategy (interpretation strategy and exploration strategy) become clearer. This must also take into account ''epistasis'', interactions among goal criteria, such as the conflict between control and evaluation dimensions. Goal criteria can be measured in an open-ended questionnaire and standardized Likert-style survey. Search landscape can be measured by evaluating a standard subset of images (i.e. a 5-by-5 neighborhood) repeatedly in different random orders, which serves to identify local maxima/minima. Creative solutions to the open-ended goals can be evaluated based on novelty (statistical unusualness) and appropriateness. Trajectories can be measured with aggregate metrics for capturing diversification/intensification: proportion of space explored, rate of exploration, instances of doubling-back, and changes in rate. The researchers are also developing a parsing language for quantifying the criterion space of creators.
They have preliminary results, mapping goals to criteria, which in turn affects the search landscape; taken together, these can be connected to diversification and intensification in one's search trajectory, as well as the evaluated outcome. They have begun to analyze goal ratings for open-ended goals on simple and complex scenes, as well as the correlations within and between users' ratings. They have begun efforts to model exploration and interpretation strategies, and are beginning to consider multiple criteria and more open-ended criteria, and how these criteria stabilize or change over time. They are considering studies to induce criteria change explicitly.
The paper concludes with relation to other work (i.e. the "//be creative//" study, in which originality of results was increased when participants were told to be creative), and a summary of limitations of their work. They acknowledge that their work addresses exploratory creativity and not transformational creativity (users do not generate anything new). They are addressing more than insight but not much more than divergent thinking, only one index (albeit a popular index) of creativity. And thus external validity is reduced, however at the gain of a depth of real-time information collected about search behaviour.
!!!!Comments & Questions
*An interesting methodology, curious to see where they take it / expand off of it, and the results they achieve
*A work-in-progress paper, results forthcoming?
*many interesting paper titles cited in the related work
*to read ref [4]: coding a semantic space - this seems tacked-on in this paper and it's not clear what value it brings to code the goal criterion in such a manner - wouldn't grounded theory apply here?
*glad they had a limitations section, notably identifying the scope of their definition of creativity.
!![Furniss2011] - on GT in HCI Research
<<cite Furniss2011>>'s 2011 CHI paper is a reflection on 3 years of research which involved the methodologies and methods of [[Grounded Theory]]. Argues that extant theory can find its place in GT analysis, in line with the evolving methodology described by <<cite Charmaz2006>>, emphasizing the need for a narrative describing processes, con-constructed with research participants, as opposed to an unstructured yet objective characterization of the major emergent themes in the data.
The paper presents a case study of 5 stages of a GT study: before the beginning, acclimatizing, fleshing out features, re-rendering the data (focus on narrative processes, rather than networks of codes / hierarchies of concepts; use of metaphor), and theoretical lenses as tools.
It also presents several lessons learnt:
#practical constraints are a reality, 'saturation' is an ideal
#open, friendly interviews
#start coding low, move upwards
#use analytical tools flexibly
#top-down constructivist approach is ok, but must match emerging, inductively-identified concepts
#appreciate multiple uses, purposes, and styles of GT
#seek participant validation
!!!!Comments & Questions
*Seems to agree with <<cite Charmaz2006>> in that extant theory can be used during analysis, but it must earn its way into your analysis by agreeing with emergent concepts.
*Top-down constructivism vs. Bottom-up / inductive objectivism (first time seeing this dichotomy)
*this paper reflects on the work that we cited the Furniss usability evaluation methods (UEM) paper in the [[DR in the wild|HDD-DR Ethnographic Project]] - use of GT / in the wild interview methodology. [[DR in the wild|HDD-DR Ethnographic Project]] was a 'technique application' GT
!![~Beaudouin-Lafon2004] - Interaction Models for Descriptive, Evaluative, Generative Power
<<cite Beaudouin-Lafon2004>>'s AVI position paper about the power of thinking about interactions rather than interfaces. Discusses interaction paradigms: computer-as-tool, computer-as-partner, computer-as-medium, as well as interaction models: instrumental interaction (describing degrees of indirection, integration, conformance), situated interaction (describing context of use, flexibility of use, reinterpretability, resilience, scalability), and interaction as a sensory-motor phenomenon. Models can be evaluated on their:
*''descriptive power'': ability to describe a range of existing interfaces
*''evaluative power'': ability to help assess multiple design alternatives
*''generative power'': ability to help designers create new designs
>//High level-models tend to have good descriptive power but poor evaluative and generative power. Low-level models tend to have poor descriptive and evaluative power, but higher generative power. A good interaction model must strike a balance between generality (for descriptive power) concreteness (for evaluative power), and openness (for generative power).//
!!!!Comments & Questions
*Well-written position paper, powers of interaction model can apply well to VA/~InfoVis (models of analysis)
!![Vicente1999]
<<cite Vicente1999>>'s book outlines [[Cognitive Work Analysis]] (CWA), a framework that combines ''constraint-based task analysis'' with ''work domain analysis'' as the foundation for a holistic and socio-technical account of an individual's work involving information technology.
''Task analysis'' has existed since the 1950s and while important part of studying work practices, it alone is insufficient for understanding these work practices or for informing design. Furthermore, most task analysis assumes an ''instruction-based approach to task analysis'', usually a ''flow-based'' or ''sequence-based approach'', in which there is a specific set of steps (or branches of steps) to follow while executing a task (flow- and sequence-based), or a set of durations associated with these steps (sequence-based, having assembly-line precision). Instruction-based approaches are fine for studying "closed-system tasks" and ''device-dependent tasks'', though what is considered to be a truly "closed system" is subjective and likely along a continuum, casting doubt over the true utility of instruction-based approaches. Despite their prevalence, these approaches are particular unsuitable for early design research.
In contrast, the underrepresented ''constraint-based approaches to task analysis'' does not outline steps but constraints on a task, and thus allows for more flexibility and worker discretion in how a task is performed, allowing for context-dependent variability and alternative strategies, such as strategies optimized for speed and strategies optimized for accuracy/precision. Strategies differ betweeen experts and novices, between experts, and between contexts. constraint-based approaches to task analysis is useful for studying "open-system tasks", those which are subject to context-dependent variability and environmental disruption. Tasks are described in a deliberately vague and abstract way to accommodate flexibility in how they are or will eventually be performed. Constraint-based approaches are ''device-independent''; this is preferable since the rationale for performing task analysis is often to improve or introduce a new design for a particular device or set of devices, a design that will change work practices upon its deployment. Rather than outline steps, these approaches place constraints within a large state space on the possibility of some steps and the order in which they might occur, on "what should //not// be done" rather than on "what must be done".
One exemplar constraint-based approach to task analysis is ''input-output'' task analysis, in which a task is specified at a high-level of abstraction according to a goal state along with its input(s) and output(s); //how// the goal state is reached is not part of the analysis, unlike instruction-based approaches to task analysis which outline each step or component of a task.
In constraint-based approaches to task analysis, a ''Strategies Analysis'' phase of CWA complements the task analysis (Rasmussen and Jensen, 1973), documenting //how// tasks are performed adaptively subject to context-dependent variability and disruptions emanating from the environment. Strategies Analysis is intended to be a systematic approach to describing strategies.
In addition, constraint-based task analysis does not answer the question of //who// does the task; as some tasks occur within a socio-technical context, tasks may be automated or performed by a human, or by a team of people working collaboratively. Later phases of CWA, namely ''Social Organization and Cooperation Analysis'' and ''Worker Competencies Analysis'', serve to answer the question of //who//.
All task analysis approaches, regardless of whether they are instruction-based or constraint-based, fall short because they are ''normative perspectives'' on work practices, on "how things should be done": tasks in which the goal state is clearly and objectively known. Task analysis does not accommodate unforeseen circumstances, error recovery tasks, or other anomalies, occasions in which exceptional work practices are carried out, nor does it explain work practices in which the goal state or outputs difficult to specify.
For this reason, task analysis is only part of the CWA framework, and work domain analysis must be performed to better understand work practices in exceptional circumstances, whereas constraint-based task analysis allows practitioners to study typical work practices and clearly defined goal states. The constraint-based task analysis used in CWA is known as ''control task analysis'' (ch. 8), and involves a method called a ''design ladder'', a modeling technique for developing control task models.
!!!!Comments & Questions
*~InfoVis EDA is an open-system task
**Input-output specification in the multi-level task typology suggests constraint-based task analysis, though our specification of //how// is problematic: this is part of ''strategies analysis'' if done descriptively, and is not typical of task analysis. We could specify that our typology of //how// is descriptive rather than normative.
**//why// and //how// as normative constraint-based task analysis - specifying a goal (why) and input and output (what), //how// as descriptive or formative strategies analysis
*See Kirwan & Ainsworth's 1992 text //A Guide to Task Analysis//, also Kugler et al (1982) as an example of task analysis
*<<cite Mullins1993>> taxonomy highly similar to Meister (1985)'s //classification of task behaviours// (sequential task analysis reference) (Table 3.1):
**''perceptual processes''
***''searching for and receiving information'': detect, inspect, observe, read, receive, scan, survey
***''identifying objects, actions, events'': discriminate, identify, locate
**''mediational processes''
***''information processing'': categorize, calculate, code, compute, interpolate, itemize, tabulate, translate
***''problem solving and decision making'': analyze, calculate, choose, compare, compute, estimate, plan
**''communication processes'': advise, answer, communicate, direct, indicate, inform, instruct, request, transmit
**''motor processes''
***''simple/discrete'': activate, close, connect, disconnect, join, move, press, set
***''complex/continuous'': adjust, align, regulate, synchronize, track
*Also see Rasmussen (1979a) on analysis of human errors, and Rasmussen (1979b) on a //morphology (taxonomy?) of mental models in man-machine contexts//
!References
<<bibliography>>
Como se pode ver em <<cite AgaGel:03 bibliography:ExampleBibliography showAll>> o exemplo n婴o feliz <<cite CasLAvRob:01>> ᠲeferᠯbrigatবt;<cite CheShaIbr:00>>, <<cite Coda:97 showAll>>, <<cite Coda:97>>, <<cite Gew:92>>,
!References
<<bibliography>>
<<bibliography ExampleBibliography showAll>>
{{{
@TECHREPORT{AgaGel:03,
author = {Agarwal, D. K. and Gelfand, A. E.},
title = "Slice Gibbs sampling for simulation based fitting of spatial data models",
institution = {University of Connecticut},
year = {2003},
}
@ARTICLE{Ait:91,
author = {Aitkin, M.},
title = {Posterior Bayes factors},
journal = {Journal of the Royal Statistical Society B},
year = {1991},
volume = {53},
pages = {111-142},
}
@MANUAL{Coda:97,
title = {CODA - Convergence Diagnosis and Output Analysis software for Gibbs sampling output: Version 0.4},
author = {Best, N. and Cowles, M. K. and Vines, K.},
organization = {MRC Biostatistics Unit},
address = {Cambrigde},
year = {1997},
}
@INPROCEEDINGS{Gew:92,
author = {Geweke, J.},
title = {Evaluating the accuracy of sampling-based approaches to calculating posterior moments},
booktitle = {Bayesian Statistics 4},
year = {1992},
editor = {Bernardo, J. M. and Berger, J. O. and Dawid, A. P. and Smith, A. F. M.},
pages = {169-193},
address = {Oxford},
publisher = {University Press},
}
@BOOK{Wol:03,
title = {The Mathematica Book},
publisher = {Wolfram Media/Cambridge University Press},
year = {2003},
author = {Wolfram, S.},
edition = {Fifth},
}
}}}
/***
|''Name:''|ExternalizePlugin|
|''Description:''|Edit tiddlers directly with your favorite external editor (html editor, text processor, javascript IDE, css editor, ...).|
|''Version:''|1.0.1|
|''Date:''|Dec 21,2007|
|''Source:''|http://visualtw.ouvaton.org/VisualTW.html|
|''Author:''|Pascal Collin|
|''License:''|[[BSD open source license|License]]|
|''~CoreVersion:''|2.1.0|
|''Browser:''|Firefox 2.0; InternetExplorer 6.0|
!Installation
#install [[it's All Text!|https://addons.mozilla.org/fr/firefox/addon/4125]] Firefox extension.
#set up [[it's All Text!|https://addons.mozilla.org/fr/firefox/addon/4125]] options in its dialog box (see tips below).
#import this tiddler from [[homepage|http://visualtw.ouvaton.org/VisualTW.html]] (tagged as systemConfig).
#save and reload.
#set up hotkey below.
#use the <<toolbar externalize>> button in the tiddler's toolbar (in default ViewTemplate) or add {{{externalize}}} command in your own toolbar.
! Useful Addons
*[[HTMLFormattingPlugin|http://www.tiddlytools.com/#HTMLFormattingPlugin]] to embed wiki syntax in html tiddlers.<<br>>//__Tips__ : When this plugin is installed, you can use anchor syntax to link tiddlers in wysiwyg mode (example : #example). Anchors are converted back and from wiki syntax when editing.//
*[[TaggedTemplateTweak|http://www.TiddlyTools.com/#TaggedTemplateTweak]] to use alternative ViewTemplate/EditTemplate for tiddler's tagged with specific tag values.
!Configuration options
|[[it's All Text!|https://addons.mozilla.org/fr/firefox/addon/4125]] extension hotkey (copy/paste from the extension dialog box)|<<option txtExternalizeHotkey>>|
|Optional tiddler containing instructions to process the text before and after externalization<<br>>Example : [[ExternalizeAsHTML]]|<<option txtExternalizeProccessing>>|
|Template called by the {{{externalize}}} button|[[ExternalizeTemplate]]|
|Max waiting time for //It's All text!// to fire|<<option txtExternalizeMaxTime>>|
!//It's all text!// extension tips
*Tiddler text is edited with the first file extension
*Copy/paste Hot Key from the dialog box (with context menu)
*Edit button isn't necessary for the plugin (it uses hotkey)
*Try the extension configuration first, before trying it with the plugin.
!Code
***/
//{{{
config.options.txtExternalizeHotkey = config.options.txtExternalizeHotkey ? config.options.txtExternalizeHotkey : "";
config.options.txtExternalizeProccessing = config.options.txtExternalizeProccessing ? config.options.txtExternalizeProccessing : "";
config.options.txtExternalizeMaxTime = config.options.txtExternalizeMaxTime ? config.options.txtExternalizeMaxTime : "30";
config.macros.externalize = {
noExtensionError : "It's all text ! extension wasn't available. Try to fire it manually with htokey or button. If it works, adapt your configuration (increase max waiting time or change hotkey) and try again.",
hotKeyError : "Hotkey wasn't understood. Use copy/paste from it's all text set up dialog.",
EmptyHotKeyError : "Hotkey isn't defined. Check ExternalizePlugin configuration.",
handler : function(place,macroName,params,wikifier,paramString,tiddler) {
var field = params[0];
var rows = params[1] || 0;
var defVal = params[2] || '';
if((tiddler instanceof Tiddler) && field) {
story.setDirty(tiddler.title,true);
var e,v;
var wrapper1 = createTiddlyElement(null,"fieldset",null,"fieldsetFix");
var wrapper2 = createTiddlyElement(wrapper1,"div");
e = createTiddlyElement(wrapper2,"textarea");
e.setAttribute("readOnly","readOnly");
v = config.macros.externalize.getValue(tiddler,field);
v = v ? v : defVal;
e.value = v;
rows = rows ? rows : 10;
var lines = v.match(/\n/mg);
var maxLines = Math.max(parseInt(config.options.txtMaxEditRows),5);
if(lines != null && lines.length > rows)
rows = lines.length + 5;
rows = Math.min(rows,maxLines);
var id=tiddler.title+"externalize"+field;
e.setAttribute("id",id);
e.setAttribute("rows",rows);
e.setAttribute("externalize",field);
place.appendChild(wrapper1);
config.macros.externalize.externalEdit(id);
return e;
}
},
externalEdit : function(id){
window.setTimeout(function(){
var element = document.getElementById(id);
if (element) {
var cpt=element.getAttribute("cpt");
cpt = cpt ? cpt -1 : parseInt(config.options.txtExternalizeMaxTime);
element.setAttribute("cpt",cpt);
if (cpt>0) {
if (element.getAttribute("itsalltext_uid")) {
element.dispatchEvent(config.macros.externalize.getKeyEvent());
addClass(element,"externalized");
}
else window.setTimeout(arguments.callee,100)
}
else alert(config.macros.externalize.noExtensionError);
}
},1000)
},
getKeyEvent : function(){
var hotkey = config.options.txtExternalizeHotkey;
if (hotkey) {
var m = hotkey.match(/^(alt)?\s*(ctrl)?\s*(meta)?\s*(shift)?\s*(\w+)\s*$/i);
if (m) {
var ev = document.createEvent("KeyboardEvent");
var cc = m[4]!=undefined ? m[5].toUpperCase() : m[5].toLowerCase();
var charCode = m[5].length==1 ? cc.charCodeAt(0) : 0;
var keyCode = m[5].length>1 ? config.macros.externalize.keyMap[m[5]] : 0;
ev.initKeyEvent("keypress",true,true,window,m[2]!=undefined,m[1]!=undefined,m[4]!=undefined,m[3]!=undefined,keyCode,charCode);
return ev;
}
else alert(config.macros.externalize.hotKeyError);
}
else alert(config.macros.externalize.EmptyHotKeyError);
},
getValue : function(tiddler,field){
var v = store.getValue(tiddler,field);
v = v ? config.macros.externalize.textProcessing(v, "Before") : "";
v = v.replace(/\[\[([^|\]]*)\|([^\]]*)]]/g,'<a href="#$2">$1</a>');
return v;
},
gather : function(e){
return config.macros.externalize.textProcessing(e.value,"After");
},
readParam : function(source,param){
var re = new RegExp("^"+ param +"\\s*: *(.*)$","mg");
var r = source && re ? re.exec(source) : null;
return r!=null ? r[1] : null;
},
textProcessing : function(text,key) {
var params = config.options.txtExternalizeProccessing;
var rexp = "^\\["+key+"\\] *(.*)\n(.*)\\n(.*)$";
if (params) {
var source = store.getTiddler(params);
source = source ? source.text : config.shadowTiddlers[params];
if (source) {
var re=new RegExp(rexp,"mg");
var instructions = source.match(re);
for(var cpt=0; cpt<instructions.length; cpt++){
re=new RegExp(rexp,"mg");
var res = re.exec(instructions[cpt]);
text = text.replace(new RegExp(res[2],res[1]),res[3]);
}
}
}
return text;
}
}
config.commands.externalize= {
text: "externalize",
tooltip: "Edit this tiddler with an external editor",
handler : function(event,src,title) {
clearMessage();
var tiddlerElem = document.getElementById(story.idPrefix + title);
var fields = tiddlerElem.getAttribute("tiddlyFields");
story.displayTiddler(null,title,"ExternalizeTemplate",false,null,fields);
story.focusTiddler(title,"text");
return false;
}
}
Story.prototype.previousGatherSaveExternalize = Story.prototype.previousGatherSaveExternalize ? Story.prototype.previousGatherSaveExternalize : Story.prototype.gatherSaveFields; // to avoid looping if this line is called several times
Story.prototype.gatherSaveFields = function(e,fields){
if(e && e.getAttribute) {
var f = e.getAttribute("externalize");
if(f){
var newVal = config.macros.externalize.gather(e);
if (newVal) fields[f] = newVal;
}
this.previousGatherSaveExternalize(e, fields);
}
}
config.macros.externalize.keyMap = {
'Backspace' : 8,
'Tab' : 9,
'Enter' : 13,
'Break' : 19,
'Escape' : 27,
'PgUp' : 33,
'PgDn' : 34,
'End' : 35,
'Home' : 36,
'Left' : 37,
'Up' : 38,
'Right' : 39,
'Down' : 40,
'Insert' : 45,
'Delete' : 46,
'F1' : 112,
'F2' : 113,
'F3' : 114,
'F4' : 115,
'F5' : 116,
'F6' : 117,
'F7' : 118,
'F8' : 119,
'F9' : 120,
'F10' : 121,
'F11' : 122,
'Num Lock' : 144,
'Scroll Lock' : 145
};
config.shadowTiddlers.ExternalizeAsHTML = "/*{{{*/\n";
config.shadowTiddlers.ExternalizeAsHTML += "[Before] g\n\\n\n<br/>\n\n";
config.shadowTiddlers.ExternalizeAsHTML += "[Before] gi\n(?:^<html>(.*)<\/html>$)|(^.*$)\n<html><body>$1$2</body></html>\n\n";
config.shadowTiddlers.ExternalizeAsHTML += "[After] g\n\\n|\\t\n\n\n";
config.shadowTiddlers.ExternalizeAsHTML += "[After] gi\n.*<html[^>]*>.*<body[^>]*>(.*)<\/body><\/html>\n<html>$1</html>\n\n";
config.shadowTiddlers.ExternalizeAsHTML += "/*}}}*/\n";
config.shadowTiddlers.ViewTemplate = config.shadowTiddlers.ViewTemplate.replace(/\+editTiddler/,"+editTiddler externalize");
config.shadowTiddlers.ExternalizeTemplate = config.shadowTiddlers.EditTemplate.replace(/macro='edit text'/,"macro='externalize text'");
config.shadowTiddlers.StyleSheetExternalize = "/*{{{*/\n";
config.shadowTiddlers.StyleSheetExternalize += ".externalized {color: [[ColorPalette::TertiaryMid]]}\n";
config.shadowTiddlers.StyleSheetExternalize +="/*}}}*/";
store.addNotification("StyleSheetExternalize", refreshStyles);
//}}}
!!!Conceptual Understanding
''FA'' is a method in which original dimensions are expressed as linear combinations of a small number of hidden or latent dimensions. The motivation often stems from the presence of clusters of dimensions that are in amongst themselves highly interrelated, but not related to other clusters.
Choosing between FA and [[PCA|Principal Component Analysis]] depends on the problem at hand. FA explains correlation while PCA accounts for variability. It must be assumed that underlying factors exist, and that they are independent of one another and of error terms. As in PCA, factors are rotated to aid interpretation. It is more likely to be successful than PCA if there is reason to believe that there is some underlying structure in the data. FA may give a more satisfactory grouping of variables.
>//"FA too closely resembles witchcraft."// - <<cite Holbrey2006 bibliography:Bibliography>> p. 41
Poorly correlated data will produce meaningless results with FA.
!!!Terminology I don't understand
*factor loadings
*communality
*specificity
*varimax rotation
*oblique projections
*canonical [variate/correlation] analysis
!!!Sources:
<<cite Holbrey2006>>
<<cite Tan2006>>
A glossary of terms:
<<list filter [tag[glossary]]>>
!Original GI work
!![Buja2009]
<<cite Buja2009 bibliography:Bibliography-GraphicalInference>> is a longer journal paper version of <<cite Wickham2010>>, without any comment on the R package implementation, and proving more examples. A larger emphasis is placed on the statistical inference technique for Exploratory Data Analysis (EDA) and Model Diagnostics (MD). It is used for qualitatively confirming discoveries and preventing over-interpretation of the data, a calibration measure to avoid missed discoveries, or preventing spurious discovery and resorting to "//pseudo-calibration of post-hoc quantitative tests tailored to the discover//".
*''EDA'' is //the free-wheeling search for structure that allows data to inform and even to surprise us// (Tukey, 1965) (before model fitting)
*''MD'': open-ended search for structure not captured by the fitted model (after model fitting)
The Rorschach method can aid an analyst in identifying the types of features that they tend to spuriously discover. The real data may be inserted with some small probability.
The line-up method results in an inferentially valid p-value. The null hypothesis is such that there's a 1 in 20 shot that the observer will single out the correct plot. Usually a feature can be described that explains why the actual data stands out, however the presence of multiple features can be beneficial or detrimental. The method can be extended to ranking protocols, and multiple observers.
Null plots are created with permutation distributions (for EDA) and residual rotation distributions (MD), //whereby random vectors are sampled in residual space with length to match the observed residual vector//.
Wording instructions is critical. Observers may have different contexts and prior knowledge about the data. Timing and confidence rating data should also be collected. Can be conducted iteratively by removing strong patterns and using the residuals to reveal fine scale patterns.
!!!!Comments & Questions
*higher emphasis on EDA was reassuring
*some statistical jargon (not a vis venue paper) - discusses how there's no need for pre-specification of discoverable figures, justification is beyond my understanding; minimally sufficient conditional sampling given a statistic (for creating a null plot); sampling a model's residual space; binomial distribution proof for calculating p-value with multiple observers in the lineup method.
!![Wickham2010]
<<cite Wickham2010>> proposes two protocols for combating ''Apohpenia'', or seeing patterns in noise. Conversely, training oneself as an analyst in these protocols may very well improve your ability to spot the "culprit" among a line-up of "innocents", that is, a visually and statistically significant result (rejecting the null hypothesis) among null plots. An R-package for generating common null plots (assuming a low-dimensionality here) and inserting the real plot is made available by the author. Null data sets are constructed by permuting a column of the data (if the null hypotheses is one in which variables are thought to be independent), or by sampling from a specified model or distribution.
>//traditional statistical tests do not cover all the complexities that arise when exploring data//
The paper discusses the use of the line-up or Rorschach protocols as a self-teaching tool.
!!!!Comments & Questions
*Does this extend to high-dimensional data - to MDS plots, or plots of principal components? Using the lineup method on scatter plots, one possible question is whether points are clustered by colour.
*Can you generate null ~SPLOMs - or will this fatigue your users? How many comparisons are necessary for graphical inference with, say, a 3x3 SPLOM? - comparing 20 plots is claimed to be reasonably feasible for a human observer.
!Citing the original GI work
!![Majumder2010]
<<cite Majumder2010>>'s Iowa State Stats. Dept. tech report presents a study of regression parameters, using the graphical inference lineup technique of <<cite Buja2009>> and making us of a subject pool on ~MTurk.
!![Zhao2012]
<<cite Zhao2012>>'s Iowa State Stats. Dept. tech report looks at <<cite Buja2009>>'s lineup technique by means of eye-tracking, comparing results to that of the earlier ~MTurk study of <<cite Majumder2010>>.
!![Wickham2011]
<<cite Wickham2011a>> describes the Ggplot2 visualization toolkit for R. An additional package, //nullabor//, generates null plots for any existing visualization created with ggplot2, for the purpose of conducting a graphical inference test.
!![Wood2011]
<<cite Wood2011>>'s TVCG paper discusses alphabetic name bias in the ordering of candidates on political ballots. They "borrow" from the process of graphical inference:
>//We borrowed from the process of graphical inference [<<cite Wickham2010>>] to compare the observed values with a null hypothesis assuming no structure to anomalies. While this indicated there might be some degree of ethnicity bias present, we wished to examine the structure of that bias in more detail.//
!![Kairam2012]
<<cite Kairam2012>>' s AVI paper discusses designing a network visualization, in which they adopt the graphical inference "//line-up//" technique with a paper prototype, 7 users:
>//We employed a methodology similar to Wickham et al.鮥-up [<<cite Wickham2010>>] for testing the 岥ntial validity�visualization technique. Specifically, we asked analysts to choose a real network data set from a line up with synthetic data using the ~GraphPrism diagrams.//
!![Ziemkiewicz2012]
<<cite Ziemkiewicz2012>>'s CHI note reports an observational study of immunologists using an immunobiology visualization, a response to the problem described in <<cite Wickham2010>> (that these users often experience apophenia in these types of visualization). The note does not report use of the graphical inference pattern perception techniques.
!On Mechanical Turk, Crowdsourcing
!![Heer2010]
<<cite Heer2010>>'s CHI paper.
!![Kosara2010]
<<cite Kosara2010>>'s BELIV paper.
!![Willett2012]
<<cite Willett2012>>'s CHI paper
!References
<<bibliography showAll>>
<<list filter [tag[graphicalInference]]>>
A qualitative pre-design evaluation technique producing rich contextual information about a process and setting in which an [[Information Visualization]] is to be used.
Qualitative data is informed by a [[Grounded Theory]] methodology, using mixed methods, until saturation is reached. This includes interviews, observations, questionnaires, artifacts produced, [[Action Research]], audio-visual materials. Thematic analysis of this data is achieved via data-driven ([[Open Coding]]), research-driven, or theory-driven coding techniques, resulting in a coherent, consistent understanding of the context.
The information acquired through this technique serves to inform the design of the future tool and also to ground subsequent evaluation during tool development and after the tool is deployed.
Source: <<cite Isenberg2008 bibliography:Bibliography>>
[>img(20%, )[http://cs.ubc.ca/~brehmer/research/grounded_theory.png]]
Notes from the <<cite Charmaz2006 bibliography:Bibliography>> reference on Grounded Theory.
!!Original Glaser & Strauss definition of GT:
*Simultaneous data collection and analysis, advancing theory generation throughout
*Highly flexible in terms of data collection methods
*Non-linear sub-processes:
**Analytic codes generated from the data, not from prior hypotheses (induced not deduced)
**The constant comparative method, comparing within and between cases, over time, across contexts
**Development of categories, memo-writing to justify / explain / rationalize / specify / relate these categories of codes
**Sampling for theory generation, not for population representativeness
*Conducting lit. review post-hoc, not beforehand
!!History of GT:
*Glaser and Strauss offer alternative to quant. techniques dominating sociology in mid 20th century w/ book //The Discovery of Grounded Theory// (1967), studying dying patients in hospitals
**Glaser (U. Columbia): the method of coding, demystifying data analysis, middle-range theories, highly empirical
**Strauss (U. Chicago): focus on the actions of individuals, not the control of higher-level structures, organizations; focus on agency, emergent processes; reliance on [[Symbolic Interactionism|Epistemology: Constructionism]], focus on language and communication, that individuals assume an active role in their actions, rather than a stimulus and response paradigm; also informed by [[Ethnography]]
*The great divide
**Glaser's later GT focuses on emergent processes, on discovery, narrow empiricism, focus on smaller social actions
**Strauss (& Corbin)'s later GT: focus on verification, theory testing, moved away from constant comparative techniques; Glaser's critique: forces data collection and analysis into preconceived categories;
*By 1990s, GT quite positivistic, used often in mixed methods research by positivist thinkers looking to triangulate with their quant. findings; however GT remains very flexible and doesn't necessarily have to be positivistic in nature;
*Charmaz's take: data and theories are not discovered, they are constructed through our ongoing experience.
!!Data collection and GT
*Mixed methods; no superior method, only contextually appropriate methods - methods as tools;
*What to initially look for / what to ask about - consider Blumer (1969)'s notion of ''sensitizing concepts'' - disciplinary perspectives, assumptions
*Criteria for assessing the quality of data gathered (p.19-20);
*Gathering GT data: big question: //"what's happening here?"// - basic social / social psychological processes occurring
*Ethnographic observations and GT - priority to the phenomenon, the process, rather than to the context, setting, organizational structure, roles actors fill (most ethnography)
*Intensive interviews - a few open-ended questions to begin, a conversation, negotiated access and rationale-buidling important; offers criteria for interviewer and participant expectations (p.26-27);
>//the researcher may have entered the implicit world of meaning, but not of explicit words//
*Texts / textual analysis - analyzing structure and content - can be primary or secondary sources of data
**Elicited texts (by the researcher: emails, questionnaires, participant diaries and journals, open-ended surveys)
**Extant texts (preexisting texts: published autobiographies, unpublished diaries and journals (unsolicited by the researcher), news articles, historical records, other published documents)
!!Coding
>//Coding is part work but it is also part play//
*Coding - select, separate, sort the data - link between collecting data and developing emergent theory
*''Initial coding'' - (open) - simple,short, and precise codes, preserve actions, compare data w/ data, move quickly through the data
**word-by-word - may be useful when structure and flow of words important, Internet ephemera
**line-by-line - maintain focus on action, look for tacit assumptions, implied actions
**incident-by-incident
**''in-vivo codes'' uses participant language, innovative terms, taken-for-granted terms,
*''Focused coding'' - using most significant / frequent codes from initial coding, for categorizing your data //incisively and completely//
*''Axial coding'' - A Strauss/Corbin term - categories and subcategories of codes - organizing schemes: conditions, actions/interactions, consequences - a frame for coding - not as flexible as focused coding (Charmaz's opinion)
*''Theoretical coding'' - sophisticated coding that follows codes from focused coding, precludes axial coding, coding families - i.e. ''Glaser's 6 C's'' - causes, contexts, contingencies, consequences, covariances, conditions - also degree, dimension, interactive, theoretical, type, identity-self, means-goals, cultural, consensus, unit, paired opposite, representation, scale, random walk, structural-functional, unit identity, temporal ordering, structural ordering, strategy, process
**Glaser's coding families very objective
**again a risk of directing analysis in a direction that allows too many preconceived notions - these codes must //earn their way into your analysis//
*In general, avoid:
**coding at too general a level
**coding topics instead of actions and processes, overlooking how people construct these
**attending to disciplinary/personal concerns rather than participants'
**coding out of context
**coding to summarize rather than analyze
*Codes bridge data with analysis - are these connections evident?
*On the benefits of coding full transcripts rather than interview field notes: good for later theoretical sampling, interview notes bears false assumption of objective transparency
!!Memo writing
!!Theoretical Sampling, Saturation, and Sorting
!!Reconstructing Theory
!!Writing and GT
!!GT studies:
*[[Grounded Evaluation]] in information visualization: (<<cite Isenberg2008>>)
*Studying collaborative use of visualization among a team of architects: <<cite Tory2008>>
*The processes of visual analysts: <<cite Kang2011>>
!!!flavours of GT (<<cite Furniss2011>>.):
*''Glaserian'': extant theory bad, inductive, emergent concepts good (highly objective)
*''Straussian'': a priori codes OK, directed questions
*''Mixed methodology'': combining GT w/ other research methods: [[Action Research]], case studies
*''technique application'': uses GT data collection and analysis methods, doesn't address construction of theory
!References
<<bibliography>>
a.k.a. DR in the Wild
<<list filter [tag[ethnoDR]]>>
An qualitative laboratory evaluation method developed by Nielsen and Molich for critiquing a system, useful for evaluating early design and detecting high-level usability problems. An inexpensive, flexible, non-intrusive, and easy-to-perform method, known as a [[Discount Usability]] technique. Heuristic Evaluation can occur at any design stage.
See [[wikipedia.org/wiki/Heuristic_evaluation|http://en.wikipedia.org/wiki/Heuristic_evaluation]].
''Method'': several (3-5) evaluators (medium expertise required) independently critique a system. Neilsen's 10 heuristics are provided, supplemented with domain- or context-specific heuristics where needed (i.e. [[Heuristics for Collaboration]]). Violations of these heuristics and their severity are recorded quantitatively:
*0 = not a usability problem
*1 = cosmetic problem
*2 = minor usability problem
*3 = major usability problem
*4 = usability catastrophe
Neilsen's 10 heuristics:
*visibility of system status
*match between system and real world
*user control and freedom
*consistency and standards
*error prevention
*recognition rather than recall
*flexibility and efficiency of use
*aesthetic and minimalist design
*help users recognize, diagnose, and recover from errors
*help and documentation
For ~InfoVis, consider the [[Visual Information-Seeking Mantra]]:
*overview first
*zoom and filter
*provide details on demand
*relate and extract information
*provide a history of interactions
In addition, there are Zuk and Carpendale (2006)'s ~InfoVis heuristics:
*ensure visual variable has sufficient length
*don't expect a reading order from colour
*colour perception varies with size of coloured item
*local contrast affects colour and grey perception
*consider colour blindness
*preattentive benefits increase with field of view
*quantitative assessment requires position or size variation
*preserve data to graphic dimensionality
*put the most data in the least amount of space
*remove the extraneous ink
*consider Gestalt laws
*provide multiple levels of detail
*integrate text wherever relevant
For domain-specific high-level tasks, heuristics based on <<cite Amar2004a bibliography:Bibliography>>'s [[Analytic Gaps]] are also appropriate:
*expose uncertainty
*concretize relationships
*determine domain parameters
*provide multivariate explanations
*formulate cause and effect
*confirm hypotheses
Conducting an evaluation:
#Be prepared - develop descriptions of a few typical tasks the system supports (specific rather than general), determine objectives and choose a set of heuristics, select expert evaluators with strong communication skills, experience conducting [[Heuristic Evaluation]], and experience with data display.
#Conduct the evaluation: have experts work independently, don't place too much emphasis on the set of heuristics, remain neutral/unbiased, take copious notes (or use video)
#Analyze the results soon after while memory of the session is clear
!!Sources
<<cite Dix2004>>
<<cite Tory2005>>
<<cite Zuk2006>>
<<cite Nielsen1990 bibliography:Bibliography-ToRead>>
>//"Any data set that with a dimensionality that is too high to easily extract meaningful relations across the whole set of dimensions. [...] dimensionality higher than 10 is considered high-dimensional."//
!!!!Source:
<<cite Bertini2011 bibliography:Bibliography>>
Welcome to my research wiki.
Here you will find a Research [[Journal]], [[Literature Review]] notes and its associated list of [[References]], [[Meeting Minutes]], a [[Glossary]], and documentation for [[Research Projects]].
<html>
<iframe src="https://www.google.com/calendar/embed?title=Matt's%20schedule&showPrint=0&showCalendars=0&mode=WEEK&height=600&wkst=1&bgcolor=%23FFFFFF&src=6mpq566c70tocji7s9he2n9c10%40group.calendar.google.com&color=%238D6F47&src=mattbrehmer%40gmail.com&color=%23AB8B00&ctz=America%2FVancouver" style=" border-width:0 " width="100%" height="600" frameborder="0" scrolling="no"></iframe>
</html>
Admin sign in here: <<option txtUserName>>
An emerging new branch of [[Human-Computer Interaction]], an information-centric approach to how humans interact with large amounts of information in external memory sources.
Source:
<<cite Pirolli2009 bibliography:Bibliography>> ch. 1 p.4, 25
//Note//: @@color:#00aa00;''MB Recommendations / Comments''@@ - please do not redistribute. Forgive typos (notes were taken on an iPad).
!!~InfoVis Session: //Ordinal & Categorical Data//
!!!!@@color:#00aa00;//~LineUp: Visual analysis of multi-attribute rankings// - <<cite Gratzl2013 bibliography:VisWeek2013 showAll>> - Best Paper Award@@
*unversity ranking use case
*drag drop sum rankings; dragable weight allocation
*interactive customized ranking and filtering; bump charts for comparing rankings
*scalability: alternative visual encoding as greyscale heatmap
*multi-objective optimization problems (MOO)
*limeup.caleydo.org, also on github
*integrating ~LineUp in ~StratomeX and Entourage Vis. Tools
*@@color:#bb0000;''Q''@@: hierarchical attributes?
*@@color:#00aa00;''MB''@@: what about deployment? User evaluation? Could this be used in Pulse for ranking buildings? Could it integrate categorical or binary variables?
*@@color:#bb0000;''Q''@@: What about ranking over time? Different time windows?
*@@color:#bb0000;''Q''@@: what about uncertainty ranking? Median rankings
!!!!//A model for structure-based comparison of many categories in small-multiple displays// - <<cite Kehrer2013 showAll>>
*small multiples and split plots; conditioning by categorical and temporal attributes; doesn't mention small multiples or faceted plots?
*comparisons of small multiples on different rows: no common baseline
*previous work uses overlays or colour comparisons to reference graphs
*comparisons across hierarchies, such as in Tableau
*absolute and relative comparisons, absolute reference specification unclear to me
*relative comparison logic requires ordinal attributes
*intermediate aggregates as derived refences; computed at different levels of the hierarchy
*car engine use case: torque example too involved, difficult to follow what's on screen
*computing intermediate aggregates over maps, comparisons over different chat types
*@@color:#bb0000;''Q''@@: how does this for time-series plots? What do intermediate aggregates look like?
*@@color:#bb0000;''Q''@@: integration with ~LineUp?
*@@color:#bb0000;''Q''@@: how does user approach the system? Where to start?
!!!!//Common angle plots as perception-true visualizations of categorical associations// - <<cite Hofmann2013 showAll>> (Iowa)
*(Skype presentation)
*Titanic survivor data: more crew survived than first class, but few people get is right. The line width problem in angle plots. Orthogonal line wish is used to make size judgments, as opposed to area in line plot, a common problem
*also seen in the Playfair import-export plot from 1786
*hammock plot (2003) encode orthogonal line width, but also runs into reverse line width illusion
*their approach: common angles, break line into three connected segments
*user study wi Titanic and gene pathway data, pairwise comparison tasks, user study had cross-over design to adjust for number of plots, tasks, users
*mappings between variables can be n:m, not necessarily n:2 as in the Titanic data
*users most correct with common angle for pairwise comparison; but no difference to hammock and parallel set plots for simple 1:1 comparisons, common angle best for ordering tasks, appears to be trickier for n:m mappings, compared space of responses in reference to illusion answers: parallel sets suffer from line-width illusion, hammock plots suffer from reverse illusion, common angle plot overcomes both
*to repeat study with more participants, data sets; common angle plot "looks" complicated
*@@color:#bb0000;''Q''@@: how does this overcome Playfair illusion and other plot types not based on parallel sets?
!!~InfoVis Session: //Perception & Cognition//
!!!!//Selecting the aspect ratio of a scatter plot based on its Delaunay triangulation// - <<cite Fink2013 showAll>>
*different aspect ratios affect contour lines and regression lines
*preserving area: clusters should become compact, trend lines should be minimized in length
*Dry et al 2009: constellations and Delauney triangulations, empirical overlap
*@@color:#00aa00;''MB''@@: alternative measures tested, did they do regression analysis or significance testing?
*select the DT with minimum edge length: naive and time consuming; alternative: continuously change scale factor and recompute DT
*user study w/ ~SPs from repository, //Nature//, participants from lab partners, 18 ~SPs
*compared against user-generated aspect ratios, compare to DT method; their method performs 59% better than users
*pre-scaled plots from. //Nature//, DT approach had smallest scaling range from original plots.
*@@color:#bb0000;''Q''@@: what approximation function? Monotone function when adjusting edge lengths in DT
!!!!//Interactive visualizations on large and small displays: The interrelation of display size, information space, and scale// - <<cite Jakobsen2013a showAll>>
*amount of zooming different between display sizes; what is impact on performance? Data and tasks differ across displays as well
*Comparing zooming, O+D, F+C interfaces between 3 display types, 2 space types (fixed information space, variable information space); 3 tasks: navigate, compare, trace tasks
*@@color:#00aa00;''MB''@@: are time results collapsed across tasks? Are there task interactions? What is criteria of Navigate task?
*avoid F*C on small displays
*larger displays don't provide benefit for multi-scale navigation, perhaps only if you normalize by size of information space
*@@color:#bb0000;''Q''@@: task complexity interaction?
*@@color:#00aa00;''MB''@@: what about fatigue and head movements for large displays?
!!!!//Hybrid-image visualization for large viewing environments// - <<cite Isenberg2013a showAll>>
*optimizing for multiple viewing distances; integrating local and global features, dynamic integration
*applying hybrid images approach from SIGGRAPH '06, frequency domain approach
*access to different spatial frequencies varies on a viewing distance curve, between 20cm and 2.5m
*envelope of visible information greater for large displays, not for desktops
*high and low pass filtering to mesh high and low detail images together
*http://aviz.fr/hybridvis examples
*visibility 쥧ibility
!!~InfoVis Session: //Defining the Design Space//
!!!!//An empirically-derived taxonomy of interaction primitives for interactive cartography and geovisualization// - <<cite Roth2013a showAll>>
!!!!//A design space of visualization tasks// - <<cite Schulz2013 showAll>>
!!!!//A multi-level typology of abstract visualization tasks// - <<cite Brehmer2013a showAll>>
!!!!//Information visualization and proxemics: Design opportunities and empirical findings// - <<cite Jakobsen2013 showAll>>
!!!!//An interaction model for visualizations beyond the desktop// - <<cite Jansen2013 showAll>>
*http://tinyurl.com/physvis
*http://aviz.fr/beyond
!!~InfoVis Session - //Systems & Sets//
!!!!//Nanocubes for real-time exploration of spatiotemporal datasets// - <<cite Lins2013 showAll>> , AT&T - Honourable Mention
*live demo, not technical so far
*algorithm walk through. me: Binning and sharing, still unclear. Roll-up cubes and time series edges, multiple spatial dimensions possible
*http://Nanocubes.net/, now available on mobile
*@@color:#00aa00;''MB''@@: handwriting font questionable
*@@color:#bb0000;''Q''@@: Shneiderman on non-sparse data, exponential growth in number of bins?
!!!!//Visualizing request-flow comparison to aid performance diagnosis in distributed systems// - <<cite Sambasivan2013 showAll>> (CMU)
*RW not automated, localize the problem, not the cause
*user study for request-flow comparison, visualizing ~DAGs
*@@color:#00aa00;''MB''@@: wordy, lots of text on slides, hard to follow, what participants saw in study unclear, what does interface look like? Ah, slide 18.
*~DAGs + diff visualization, before and after DAG, animation; result: no single best approach
*is this a design study?
*@@color:#bb0000;''Q''@@: is this deployed? No.
!!!!@@color:#00aa00;//Evaluation of filesystem provenance visualization tools// - <<cite Borkin2013a showAll>> @@
*node-link diagrams or radial layouts? For Filesystem provenance data
*IT support scenarios
*~InProv design study, radial layout, context views for visual history, time-based grouping, looks for natural groupings of activities
*quantitative study or radial vs. node-link (within subjects), datasets size (within subjects), time-based vs. conventional node grouping (between subjects), expert user participants, accuracy, efficiency, subjective workload measures, real Filesystem provenance data from existing data challenge repositories, questions/tasks adapted from those challenges
**results: for larger datasets, participants more efficient, not more accurate for conventional grouping; for hard data and time-based grouping, participants more accurate
**women less accurate for node-link diagrams? Gender HCI, known low-level perceptual and cognitive differences between mean nod women and spatial tasks
*@@color:#00aa00;''MB''@@: great presentation, slides, pacing
*http://bit.ly/InProv, http://bit.ly/mborkin
*@@color:#bb0000;''Q''@@: enough to generalize about gender differences? Yes it was significant
*confidence varied based in grouping encoding
*@@color:#bb0000;''Q''@@: Grinstein on experimenter-participant gender interaction may have confounded gender difference
*node grouping differences occurred for both layouts
*@@color:#bb0000;''Q''@@: no task-dependent differences, tasks weren't exploratory
!!!!//Visualizing fuzzy overlapping communities in networks// - <<cite Vehlow2013 showAll>>
*community overlaps, RW on disjoint or crisp communities, not fuzzy overlapping communities
*pie glyphs as vertices, nodes and meta nodes, uncertainty and fuzzy community membership, pie glyphs angled toward meta-nodes to which it has fuzzy membership
*@@color:#00aa00;''MB''@@: math-heavy / algorithm focused; what did interface look like
*case study in protein-interaction networks
*FW: focus+context techniques; optimize colour; user study, case studies in other areas
*@@color:#bb0000;''Q''@@: social network analysis? Or co-citation networks, research overlaps.
*@@color:#bb0000;''Q''@@: aggregating the data, what is threshold? Fuzzyness normalized, user-defined thresholds
!!!!//Radial sets: Interactive visual analysis of large overlapping sets// - <<cite Alsallakh2013 showAll>>
*limited scalability of Euler diagrams
*radial layout with degree of overlap as proximity to centre of layout, edges between sets encode overlap; set positions and size to support scalability, overlaps could be absolute or relative to size of sets, could also colour overlaps to show significance of overlap
*overview first; details on demand (Ben S. Floating head)
*@@color:#00aa00;''MB''@@: lightning fast demo
*overlap size encoding as circles in centre; doesn't show connections to sets to avoid clutter, overlap connections only shown on interaction/brushing
*visual complexity doesn't afford walk-up and use, requires training, but user studies apparently show
*http://radialsets.org
*@@color:#bb0000;''Q''@@: what tasks? How do sets overlap? How do elements belong to sets? How is an attribute distributed?
*@@color:#bb0000;''Q''@@: dates, numerical sets? Maybe?
*@@color:#bb0000;''Q''@@: visual complexity and order of set layout? Algorithm in paper, a hard optimization problem, in paper.
!!~InfoVis Session - //Application Areas//
!!!!//~SoccerStories: A kick-off for visual soccer analysis// - <<cite Perin2013a showAll>> - Honourable Mention
*realtime, not aggregated sport data. different use cases, user types
*temporaland spatial contexts, important not to lose this overview and detail
*small multiples; faceted views on demand on the focus view, mode-link, adjacency matrix, tag cloud
*@@color:#00aa00;''MB''@@: tool walkthrough is not tied to tasks, too descriptive, fast
*game phases in small multiple views, difficult to tell what is going on with transformations
*analysts use the tool for several months, including sports journalists
*sportlines, small embedable visualizations to place in the body text of an article
*@@color:#bb0000;''Q''@@: Stasko: what defines a phase? The events extracted by an algorithm; offensive events, e.g. A shot on goal, considering series of events before and after shot
!!!!//Understanding interfirm relationships in business ecosystems with interactive visualization// - <<cite Basole2013 showAll>>
*@@color:#00aa00;''MB''@@: blah blah blah BI buzzword bingo, no text on slides
*which firms work with whom? Market research domains
*SDC platinum, Reuters source tool; relationships b/w firms: strategic, supply chain, manufacturing, R&D, licensing; table format, difficult to query; also queries compustead on financial data
*expert interviews, field study, lit review; patent lawyers, analysts
*~DotLink: 7 interconnected views, overview, details,crabs, lists, radial layouts, linked views, linked selection: time, space, connections
*FW: longitudinal user study
*@@color:#00aa00;''MB''@@: user deployment? Evaluation?
*@@color:#bb0000;''Q''@@: Endert: how does this fit in w/ analysts' workflow? Great for quick reference, especially for consultants diving into a project in an unfamiliar domain; only part of a workflow
!!!!//Creative user-centered visualization design for energy analysts and modelers// - <<cite Goodwin2013a showAll>>
*@@color:#00aa00;''MB''@@: could creativity methods extend to remote target users? How many users do you need?
*didn't cover post-deployment stages, continued usage
!!!!@@color:#00aa00;//Entourage: Visualizing relationships between biological pathways using contextual subsets// - <<cite Lex2013 showAll>> @@
*relationships between pathways and drug interactions, tabular data; a need to integrate
*contextually-relevant pathways related to focus pathway
*inspired by parallel tag clouds by Collins et al. 2009
*multiple pathways, pathway relationships, integration with enRoute [Partl et al 2012], showing associated experimental data, unable to show cycles in pathways
*@@color:#bb0000;''Q''@@: deployment? Evaluation? Actual user or hypothesized user; a case study researcher at Novartis, deferred case study detail referred to in paper
*http://entourage.caleydo.org
*context-preserving visual links
!!!!//Variant View: Visualizing sequence variants in their gene context// - <<cite Ferstay2013 showAll>>
!!~InfoVis Session - //Time, Trees & Graphs//
!!!!@@color:#00aa00;//Visualizing change over time using dynamic hierarchies: ~TreeVersity2 and the ~StemView// - <<cite Guerra-gomez2013 showAll>> @@
*nice summary of types of tree comparison - aggregatable and hierarchical trees; e.g. Budgets of govt agencies vs. salaries of employees
*The Bullet (Gomez et al. 2012)
*colour and size encoding, horizon lines, change summary reporting tool
*live demo; well done. Likely not colour-blind safe, but colour is redundant anyway
*13 case studies with 9 organizations / agencies
*EBGM: empiric Bayes geometric means
*http://Treeversity.cattlab.umd.edu
!!!!//Visual compression of workflow visualizations with automated detection of macro motifs// - <<cite Maguire2013 showAll>>
*macros in electrical circuit diagrams as analog
*represent an experiment as a workflow for reproducibility, ~AutoMacron uses motifs that preserve workflow topology
*@@color:#00aa00;''MB''@@: hard to follow this talk, presented very fast, a lot going on in the sliders
*@@color:#00aa00;''MB''@@: who are domain users? What domain?
*evaluation is unfair since this is a new technique?
*part of ~IsaCreator, open source, used in biological sciences
*http://GitHub.com/isa-tools/AutoMacron
!!!!//Automatic layout of structured hierarchical reports// - <<cite Bakke2013 showAll>> , (MIT CSAIL)
*structured forms to avoid database view ~UIs
*live demo
*nested tables and forms, not space efficient
*automatic column adjustments based on variable-width columns, to ensure no horizontal scrolling and no wasted space; previously would have been done manually, optimal placement
*demo ran over time, rushed through all the presentation, no conclusions, apparently a user study
!!!!@@color:#00aa00;//Edge compression techniques for visualization of dense directed graphs// - <<cite Dwyer2013 showAll>>@@
*example: dependencies in source code - simplifying dense graphs to a style familiar to software engineers used to block diagrams
*evaluation: path following tasks on flat graphs vs compressed graphs with modular decomposition
*results: compression helps with efficiency, not accuracy
*another study examining the effect of boundary crossings in path tracing task
*~MiniZinc contstraint programming language to minimize crossings
*edge compression works for path following in dense directed graphs but you have to constrain the number of crossings, allowing for more compression
*@@color:#00aa00;''MB''@@: of potential interest to JD re: undirected graphs? Boundary crossings as path tracing difficulty factor, together with edge-edge crossings, edge-node crossings
*@@color:#bb0000;''Q''@@: readability for very large flat graphs?
!!VAST Session - //Text and Social Media//
!!!!//~HierarchicalTopics: Visually exploring large text collections using topic hierarchies// - <<cite Dou2013a showAll>>
*@@color:#00aa00;''MB''@@: of potential interest to SI
*~ParallelTopics Dou et al 2011, scalability Issues w/ number of topics
*visual metaphor and interaction exploration for topic exploration
*~TopicRoseTree algorithm: join, absorb, collapse; determine cost of merging two trees, calculating probablity distribution of topics over full vocabulary of corpus
*different users have different user models, so users can modify the automatically generated tree using the same three steps: join, absorb, collapse
*@@color:#00aa00;''MB''@@: pros/cons of user join, absorb, collapse vs. tagging in Overview, how can users annotate HT, see paper?
*case study of NSF programs' impact on research, follow-up with web searches to understand trends
*user study with CNN corpus; comparing hierarchical vs. non-hierarchical interfaces; comparing number of topics identified, time to identify topics
*users prefer flat topic list over nested hierarchy, splitting topics needed, but hierarchical was faster and more scalable
*@@color:#bb0000;''Q''@@: Enamul (UBC): when to allow users to modify? Computational overhead? Only at interface level, not at weighting of algorithm.
*@@color:#bb0000;''Q''@@: documents in multiple topics: duplicating documents!
!!VAST Session: //~High-Dimensional Data//
!!!!@@color:#00aa00;//Explainers: Expert explorations with crafted projections// - <<cite Gleicher2013a showAll>> - Honourable Mention@@
*@@color:#00aa00;''MB''@@: of potential interest to SI
*comedic-ness of Shakespeare's plays, user-generated weighings
*organize and explain; using understandable variables; must give control and tradeoff of understandability and expressiveness
*~SVMs used to implement explainers, sorting and filtering of many ~SVMs, very much like scagnostics
*goal of explainers: organize data according to user-defined concepts, explain user-defined concepts according to data
*semantic interaction et al. doesn't allow for the explanation of user-defined concepts according to data
*DR will organize data according to statistics, but not according to user-defined concepts
*http://graphics.cs.wisc.edu/Vis/explainers
*@@color:#bb0000;''Q''@@: what if concepts aren't actually in the data? Explainers could be first step before significance testing; with DR, you begin with significance testing, then find out if results are interesting, with Explainers, you find what's interesting, the do significance testing
*@@color:#bb0000;''Q''@@: multiple competing explainers; you can throw out explainers you don't care about
!!!!//Semantics of directly manipulating spatializations// - <<cite Hu2013 showAll>>
*@@color:#00aa00;''MB''@@: of potential interest to SI
*algorithm for reverse-engineering the user's guidance of the spatialization
*when you move point A from group C to to up B, is A different from C or similar to. B, an ambiguity arises; previous implementations didn't consider unmoved objects
*bias example with groups of different sizes; weighting to reduce bias
*Virginia census data example, 134 data points, moving the small college towns away from the other cities
*@@color:#00aa00;''MB''@@: difference from semantic interaction? Explainers?
*@@color:#bb0000;''Q''@@: Alex Endert: on mathematical elegance vs. expressiveness and understandability? @@color:#bb0000;''A''@@: provide user w/ default setting for controlling the spatializations, the C{I,j} weighting factor
*@@color:#bb0000;''Q''@@: scalability: demo data set was very small
!!!!//~SketchPad ~N-D: WYDIWYG sculpting and editing in + LJK dimensional space// - <<cite Wang2013a showAll>>
*how to generate a high-dimensional data set? Alternatives to coding?
*sketching high-dim parallel coordinates and SPLOM plots, begin with random/ default data set
*user study: given a target plot, create a sketch to match
*sketching scatter plots, sculpting a point cloud, add fuzziness
*@@color:#00aa00;''MB''@@: can you sketch clusters or just trends? Speaking too fast, can't follow
*data generation and data visualization at the same time; sketching a data set
*@@color:#bb0000;''Q''@@: don't start from random data, start from distributions? @@color:#bb0000;''A''@@: you can start from existing data, or start from random data; Q: how to draw N-dimensional correlations in parallel coordinates? @@color:#bb0000;''A''@@: reorder the axes, draws PDF again
!!!!//Visual analysis of higher-order conjunctive relationships in multidimensional data using a hypergraph query system// - <<cite Shadoan2013 showAll>>
*(M.Sc student)
*example: which European country produced the most mistresses during the enlightenment?
*Improvise, Candid, and the electronic enlightenment data set; high-dim categorical and ordinal data
*@@color:#00aa00;''MB''@@: use annotation/ highlighting on the slides; I can't follow what part of the interface you're referring to, update: her slides weren't even shown, interface was shown the entire time!
*live demo of. Candid, diplomats, clergymen to politicians, hyper edge queries
*compact way of recording queries;
*no user studies yet; will it work with views other than graphs?
*@@color:#bb0000;''Q''@@: tool cannot import other data sets, but hypergraph techniques generalize to other data sets
!!!!//Interactive exploration of implicit and explicit relations in faceted datasets// - <<cite Zhao2013 showAll>>
*first author Zhao unable to present due to visa issues and govt shutdown, presented by Collins
*Document data; code available at http://cs.toronto.edu/~jianzhao
*RW: building on Polaris, ~InfoZoom, semantic substrates, pivot graphs
*~PivotSlice: visual query language for and/or queries
*example and live demo: graphs of papers, citations, papers citing it, authors, keywords, years, venues; Faceting and collapsing, overview, pivot, slice, relate, extract, details-on-demand
*qualitative evaluation with grad students, used Amar and Stasko task taxonomy for prescribed tasks, FW to include a deployment study, more datasets
*http://tiny.cc/PivotSlice
*@@color:#bb0000;''Q''@@: duplicates in queries
!!VAST Session - //Sensemaking and Collaboration//
!!!!//Supporting awareness through collaborative brushing and linking of tabular data// - <<cite Hajizadeh2013 showAll>>
*(presented by Tory)
*how should brushing actions be visible across remote and/or asynchronous collaboration?
*how would this interfere with one' sown brushing?
*Isenberg and Fisher (2009) - Cambiera, for co-located collaboration, brushing and linking for awareness; not for analysis; Tory et al take opposite stance
*goal: maximize awareness while not interfering with individual awareness
*comparing selection; persistent selection; brushing and linking, control, for awareness.
*"collaborator" was a script, unbeknownst to participant
*perceived expert status of collaborator
*still haven't applied to practical deployed situations?
*@@color:#00aa00;''MB''@@: teams of collaborators? Applicability of cooperative gaming and tabletop collaboration research
*@@color:#bb0000;''Q''@@: lack of realism: no voice, voice is essential for collaboration
!!!!//Identifying redundancy and exposing provenance in crowdsourced data analysis// - <<cite Willett2013 showAll>>
*extends CHI '12 paper: rate clarity, specificity, check sources, provenance, remove redundancy
*cross-domain security model limits understanding of source allocation to participant responses; solution, uses an embedded browser, produce detailed logs of browsing activity, track highlights
*allow crowd workers to perform clusterings to eliminate redundancies; but difficult to combine clusterings; find a single clustering most representative of the group of clusterings, close to subject matter expert clustering
*analyst still has to filter and explore explorations
*@@color:#bb0000;''Q''@@: citizen science connection? Reward structures, perhaps a paid structure
*FW: domain workers with more training
*@@color:#bb0000;''Q''@@: diversity vs. refinement
*use case: journalist as analyst
!!!!@@color:#00aa00;//The impact of physical navigation on spatial organization for sensemaking// - <<cite Andrews2013 showAll>> @@
*head movements, eye movements, body movement
*evaluation with two display conditions, 2 hour session with 58 intelligence reports
*analyst's workspace - VAST 2011
*no overall difference between groups using large and small displays, but spatial organization of documents varied tremendously, as did performance measures
*attempted geospatial organization was not helpful, concept map approach was best, no organization scheme also bad
*tracking document movements, examined spatial arrangement
*large display users created more structures based on Gestalt principles, complex and simple structures, 30% of structures in large display were considered complex
*small display users seldom used viewport in overview+detail
*eyetracking didn't work very well for large display users
*also examined note-taking behaviour, labels and narrative notes and their placement, either labelling the space or labelling documents they were near; large display users created narrative nots, small display users made label notes, large display users externalized more
*small display users didn't use overview for structure, only for navigation
*changing display environment change high-level behaviour
*@@color:#bb0000;''Q''@@: (Enamul, UBC) individual abilities? Spatial ability? No, FW.
*@@color:#bb0000;''Q''@@: inexperienced analysts: what about pro analysts? @@color:#bb0000;''A''@@: we all organize things spatially, regardless of experience
*@@color:#bb0000;''Q''@@: (W. Wong): were structures qualitatively different and how did they correlate with success? @@color:#bb0000;''A''@@: no; is there upper bound on display size? @@color:#bb0000;''A''@@: haven't found it yet
!!!!//Using interactive visual reasoning to support sense-making: Implications for design// - <<cite Kodagoda2013 showAll>> , Middlesex
*(presented by Kodagoda and Wong)
*@@color:#00aa00;''MB''@@: buzzword-laden title; Ugh, text-heavy slides
*Invisique tool (http://Invisique.com), tactile reasoning, distributed cognition, index-card sorting metaphor
*Takken and Wong (proc NDM '13), card-sorting
*user study with tmm aloud; screen capture, user observation, SMART probes (cognitive task analysis), task: find most influential infovis authors in in SIGCHI corpus
*data-frame model of sense making
*naive users in evaluation
*implications for design: me: how is this novel? What is contribution?
*@@color:#bb0000;''Q''@@: Vicki Lemieux: what about heterogeneous documents? @@color:#bb0000;''A''@@: dunno; We'll use same protocol.
*@@color:#bb0000;''Q''@@: handwritten notes from subjects? @@color:#bb0000;''A''@@: will incorporate that.
!!VAST Session - //Temporal Analytics//
!!!!@@color:#00aa00;//Temporal event sequence simplification// - <<cite Monroe2013b showAll>> - Honourable Mention@@
*~EventFlow - compact views to summarize large temporal event datasets, aggregation of records and events
*focus on simplifying the underlying data
*search and replace functionality, event simplification and replace, filtering, reduce the numb of elements, increase the size of remaining elements
*live demo
*rainbow colour palette; (Customizable)
*@@color:#bb0000;''Q''@@: replace specificity: see CHI'13 paper for search query language
*@@color:#bb0000;''Q''@@: indications of how many patterns replaced? Yes
*@@color:#bb0000;''Q''@@: errors, temporal uncertainty, granularity, missing data
!!!!//Visual analytics for model selection in time series analysis// - <<cite Bogl2013 showAll>>
*~Box-Jenkins methodology for finding models in time-series data, previously cumbersome, lots of scripting
*ARIMA models; process follows Lammarsch et al. (2011), interactive visual support for finding these models, immediate visual feedback of model residuals
*~TiMoVA prototype - residual analysis, alpha blending, residuals should be white noise
*Java, Prefuse, ~TimeBench, R
*video, demo, not enough guidance, annotation
*usage scenarios and example dataset, formative evaluation
*FW: deploy as R package
*@@color:#bb0000;''Q''@@: comparison of multiple models? Not side by side?
*@@color:#bb0000;''Q''@@: univariate time series, stationary, seasonal behaviour, bin sizes provided by time series
!!!!//~TimeBench: A data model and software library for visual analytics of time-oriented data// - <<cite Rind2013 showAll>>
*challenges of time-oriented data, summarized by Aigner et al 2011, granularity and granule summary
*~TimeBench components: calendar operations, data structures, data transformations, visualization and interaction, based on Prefuse toolkit, follows Heer and Agrawala software design patterns for visualization
*temporal indeterminacy, different time primitives, multiple granularities and cycles
*~PlanningLines demo with uncertain starts and ends of intervals
*GROOVE example - Lammarsch et al. (2005)
*evaluation: application examples, long-term developer studies, student projects and research projects, used in ~TiMoVA, high-school student projects
*results: expressiveness, common data structure, developer accessibility, runtime efficiency
*available on github, http://TimeBench.org
*@@color:#bb0000;''Q''@@: scalable? @@color:#bb0000;''A''@@: Deals with temporal complexity, still may not be scalable
*@@color:#bb0000;''Q''@@: data structures based on Prefuse
*@@color:#bb0000;''Q''@@: time zone switching? Absolute vs. relative time, planned for FW
!!!!//~MotionExplorer: Exploratory search in human motion capture data based on hierarchical aggregation// - <<cite Bernard2013 showAll>>
*motion capture data, macro and micro scales, a design study
*requirements analysis: paper prototyping: overview, steerable aggregation, filtering, hierarchical clustering of poses
*human motion pose clustering, motion sequence exploration, colour coding by motion type
*formative lab design study conducted with 14 non-experts
*summative insight-based field study with 5 domain experts: prescribed loosely-defined tasks: become confident with system, identify a specific pose, perform a pose sequence query, identify style variation; identify cyclic behaviour, such as walking or cartwheels
*FW: multi variate time oriented data, state transition data
*@@color:#bb0000;''Q''@@: interactive aggregation level change; how does this change semantics of motion?
*@@color:#bb0000;''Q''@@: why just first 2 PCA components? Only shows coarse, macro movements, not fine-grained motion
!!!!//Supporting the visual analysis of dynamic networks by clustering associated temporal attributes// - <<cite Hadlak2013 showAll>>
*icicle plots and attribute-specific time-series plots
*deployment with domain experts, discovering clusters, global and local levels; user studies FW
!!Panel Session - //The Role of Visualization in the Big Data Era: An End to a Means or a Means to an End?//
!!!!Danyel Fisher (MSR)
*visualization for exploration as means to an end
*visualization for presentation as an end to the means
*guided exploration in interactive journalism
!!!!Carlos Scheidegger (AT&T)
*visualization is a means to a means
*"technology exists to solve problems created by technology"
*protovis vs. d3: d3 is wildly popular, why?
*the complexity ratchet; we need more domain-specific languages.
*don't impose structure on the data, model-freeness
*rcloud: http://GitHub.com/csheid/rcloud
*slides: http://bit.ly/VisWeek-bigdata-2013
!!!!Heidi Lam (Google)
*analysis and display, contrasting, continuum, or loop?
*how biased is the overview?
*challenge 1: figure out what NOT to show
*Simpson's paradox, aggregated values may be misleading
!!!!Robert Kosara (Tableau)
*you're doing it all wrong
*big data is real; it's big, it lives in databases
*Hitchhiker's Guide quote
*go to where the data is? Tableau imports from files and 35 database types; ~InfoVis prototypes only import from text files
*Fekete / Plaisant ~InfoVis 2002 - interactive viz of a million items
*we can't throw pixels at the problem, LDAV best paper showed that people don't get value out of large displays
!!!!Daniel Keim (U. Konstanz)
*Visualization of big data does not work
*visual information seeking mantra doesn't work
*visual Analytics paradigm instead: analyze first, show important, zoom, filter, analyze, details on demand
*example: Google Analytics works, but doesn't leverage visualization
*automated analysis of big data works (only under certain preconditions, preconditions are rarely met, and problem rarely well-defined)
*visualization is still needed; interaction and steering of automated analysis
!!!!Discussion
*@@color:#bb0000;''Q''@@: Scheidegger@Keim: what so you mean it doesn't work?
*@@color:#bb0000;''Q''@@: Kosara@Keim: you overestimate the usefulness of machine learning
*@@color:#bb0000;''Q''@@: Fekete: big data and Vis requires too much prerequisite knowledge for ~PhD students; Fisher: artificial boundaries of academia
*@@color:#bb0000;''Q''@@: "Big Data": two short words that congress can understand - Chris Johnson
*@@color:#bb0000;''Q''@@: Chris North: where's the cognitive psychology in this discussion? How can we think about exabytes?
!!Workshop - //Mobile and Breaking News: New Challenges for News Information Visualizations//
*[[Nick Diakopoulos|http://www.nickdiakopoulos.com/]] - //SRSR//: tweet credibility for breaking news - ([[Finding and assessing social media information sources in the context of journalism|http://dl.acm.org/citation.cfm?doid=2207676.2208409]] - Diakopoulos et al. Proc. CHI '12)
*[[Deok Gun Park|http://www.intuinno.com/about-me.html]] - [[VizTwit|http://newsvis.com/wp-content/uploads/2013/10/newsvis13-deok.pdf]] - localized tweets on news visualizations, dilemmas and approaches, Bostock's NYT interactive visualizations re: //512 paths to the White House//, //a constellation of directors and their stars//, paradox of choice, credibility and spam detection
*[[Ramik Sadana|http://ramiksadana.com/]] - ([[TouchViz: a case study comparing two interfaces for data analytics on tablets|http://dl.acm.org/citation.cfm?doid=2470654.2481318]] - Drucker et al. Proc. CHI '13 MSR on gestures for mobile / tablet viz, challenges for charts other than bar charts
*[[Sara Quinn (Poynter Institute)|http://about.poynter.org/about-us/our-people/sara-quinn]] - [[eyetracking and gestures tracking for mobile / tablet news sites / layouts|http://www.poynter.org/how-tos/newsgathering-storytelling/visual-voice/191875/new-poynter-eyetrack-research-reveals-how-people-read-news-on-tablets/]], carousel design/layout preferred but had high variance in response
!!Keynote - //Visualization vs. Expertise: Case Studies in Lexicography and Genomics//
*Speaker: Erez Lieberman Aiden, Assistant Professor Department of Molecular and Human Genetics, Baylor College of Medicine Department of Computer Science, Department of Computational and Applied Mathematics, Harvard Society of Fellows Director, The Center for Genome Architecture
*equilibrium and fractal globules of chromosomes, ramen noodle analogy, tangled slinkies
*facebook photo analogy for understanding interactions in genomes; cluster analysis and correlation analysis, plaid carpet patterns of patterns and anti-patterns, open and closed chromatins
*collab w/ google to combine dictionaries and visualization, digitized book collections
*n-grams and cultural trends, thrived vs. throve, unforgettable 1950
*//Uncharted// book out this December
!!Capstone - J. van Wijk: //Information Visualization: Challenges and Opportunities//
*custom point solutions the only answer? Unavoidable complexity? Need more powerful tools?
*~MagnaView had too many options
*what does the data want to be?
*spectrum from business graphics to ~InfoVis to VA, based on dataset size; move toward the left, reduce size with filtering, data, statistics, machine learning, but mustn't lose critical information
*Anscombe's quartet, example of loss of information
*importance of understanding perception
*Maureen Stone's In [[Color Perception, Size Matters|http://dx.doi.org/10.1109/MCG.2012.37]] paper, CG&A 2012
*Holtem and van Wijk 2009 on edge direction encodings
*lesson 1: evaluation is crucial; lesson 2: avoid evaluation
*Dieter Rams, industrial design, Braun, ten principles of good design
*Synerscope - edge bundling
*van Wijk and Nuij (2003) on optimal model based visualization
*grand tour of ~PCPs: Holten and van Wijk (2010); mathematically clean 峡ble
*real world as inspiration, example: botanically-inspired tree visualization
*bipolar examples: pessimist and optimist perspectives on visualization
*Think as a user, be a user, act as a user (not as a reader, reviewer)
*perspective on companies like Tableau, Spotfire; they keep it simple and that's how they are successful
*simplicity toward complexity vs. complexity toward simplicity; the latter
*reviewing strategy: look at the pictures, then the captions, then the text
*@@color:#bb0000;''Q''@@: are we engineers? Scientists? We are designers.
*@@color:#bb0000;''Q''@@: how to get insight out of a visualization? Getting insights is the holy grail. Build up from searches, queries, bottom-up
*@@color:#bb0000;''Q''@@: cognitive bias risk? Designers can get locked into bias of the user as well; everyone has their own biases; don't us your own hammer for every nail
*@@color:#bb0000;''Q''@@: role of education: how to teach visualization? Awareness, to view and watch visualization critically as a first step, showing good examples
!!Additional Notes:
[[Google Doc Notes by Skau, Pyzh, MacNeil, Harrison, Alsallakh|https://docs.google.com/document/d/13eEEO8ZXy-vGajBsG6iMQBAwp3z6agtxtUdEGceDfcQ/edit#]]
!!References:
<<bibliography showAll>>
/***
|Name|ImageSizePlugin|
|Source|http://www.TiddlyTools.com/#ImageSizePlugin|
|Version|1.2.3|
|Author|Eric Shulman|
|License|http://www.TiddlyTools.com/#LegalStatements|
|~CoreVersion|2.1|
|Type|plugin|
|Description|adds support for resizing images|
This plugin adds optional syntax to scale an image to a specified width and height and/or interactively resize the image with the mouse.
!!!!!Usage
<<<
The extended image syntax is:
{{{
[img(w+,h+)[...][...]]
}}}
where ''(w,h)'' indicates the desired width and height (in CSS units, e.g., px, em, cm, in, or %). Use ''auto'' (or a blank value) for either dimension to scale that dimension proportionally (i.e., maintain the aspect ratio). You can also calculate a CSS value 'on-the-fly' by using a //javascript expression// enclosed between """{{""" and """}}""". Appending a plus sign (+) to a dimension enables interactive resizing in that dimension (by dragging the mouse inside the image). Use ~SHIFT-click to show the full-sized (un-scaled) image. Use ~CTRL-click to restore the starting size (either scaled or full-sized).
<<<
!!!!!Examples
<<<
{{{
[img(100px+,75px+)[images/meow2.jpg]]
}}}
[img(100px+,75px+)[images/meow2.jpg]]
{{{
[<img(34%+,+)[images/meow.gif]]
[<img(21% ,+)[images/meow.gif]]
[<img(13%+, )[images/meow.gif]]
[<img( 8%+, )[images/meow.gif]]
[<img( 5% , )[images/meow.gif]]
[<img( 3% , )[images/meow.gif]]
[<img( 2% , )[images/meow.gif]]
[img( 1%+,+)[images/meow.gif]]
}}}
[<img(34%+,+)[images/meow.gif]]
[<img(21% ,+)[images/meow.gif]]
[<img(13%+, )[images/meow.gif]]
[<img( 8%+, )[images/meow.gif]]
[<img( 5% , )[images/meow.gif]]
[<img( 3% , )[images/meow.gif]]
[<img( 2% , )[images/meow.gif]]
[img( 1%+,+)[images/meow.gif]]
{{tagClear{
}}}
<<<
!!!!!Revisions
<<<
2011.09.03 [1.2.3] bypass addStretchHandlers() if no '+' suffix is used (i.e., not resizable)
2010.07.24 [1.2.2] moved tip/dragtip text to config.formatterHelpers.imageSize object to enable customization
2009.02.24 [1.2.1] cleanup width/height regexp, use '+' suffix for resizing
2009.02.22 [1.2.0] added stretchable images
2008.01.19 [1.1.0] added evaluated width/height values
2008.01.18 [1.0.1] regexp for "(width,height)" now passes all CSS values to browser for validation
2008.01.17 [1.0.0] initial release
<<<
!!!!!Code
***/
//{{{
version.extensions.ImageSizePlugin= {major: 1, minor: 2, revision: 3, date: new Date(2011,9,3)};
//}}}
//{{{
var f=config.formatters[config.formatters.findByField("name","image")];
f.match="\\[[<>]?[Ii][Mm][Gg](?:\\([^,]*,[^\\)]*\\))?\\[";
f.lookaheadRegExp=/\[([<]?)(>?)[Ii][Mm][Gg](?:\(([^,]*),([^\)]*)\))?\[(?:([^\|\]]+)\|)?([^\[\]\|]+)\](?:\[([^\]]*)\])?\]/mg;
f.handler=function(w) {
this.lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = this.lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
var floatLeft=lookaheadMatch[1];
var floatRight=lookaheadMatch[2];
var width=lookaheadMatch[3];
var height=lookaheadMatch[4];
var tooltip=lookaheadMatch[5];
var src=lookaheadMatch[6];
var link=lookaheadMatch[7];
// Simple bracketted link
var e = w.output;
if(link) { // LINKED IMAGE
if (config.formatterHelpers.isExternalLink(link)) {
if (config.macros.attach && config.macros.attach.isAttachment(link)) {
// see [[AttachFilePluginFormatters]]
e = createExternalLink(w.output,link);
e.href=config.macros.attach.getAttachment(link);
e.title = config.macros.attach.linkTooltip + link;
} else
e = createExternalLink(w.output,link);
} else
e = createTiddlyLink(w.output,link,false,null,w.isStatic);
addClass(e,"imageLink");
}
var img = createTiddlyElement(e,"img");
if(floatLeft) img.align="left"; else if(floatRight) img.align="right";
if(width||height) {
var x=width.trim(); var y=height.trim();
var stretchW=(x.substr(x.length-1,1)=='+'); if (stretchW) x=x.substr(0,x.length-1);
var stretchH=(y.substr(y.length-1,1)=='+'); if (stretchH) y=y.substr(0,y.length-1);
if (x.substr(0,2)=="{{")
{ try{x=eval(x.substr(2,x.length-4))} catch(e){displayMessage(e.description||e.toString())} }
if (y.substr(0,2)=="{{")
{ try{y=eval(y.substr(2,y.length-4))} catch(e){displayMessage(e.description||e.toString())} }
img.style.width=x.trim(); img.style.height=y.trim();
if (stretchW||stretchH) config.formatterHelpers.addStretchHandlers(img,stretchW,stretchH);
}
if(tooltip) img.title = tooltip;
// GET IMAGE SOURCE
if (config.macros.attach && config.macros.attach.isAttachment(src))
src=config.macros.attach.getAttachment(src); // see [[AttachFilePluginFormatters]]
else if (config.formatterHelpers.resolvePath) { // see [[ImagePathPlugin]]
if (config.browser.isIE || config.browser.isSafari) {
img.onerror=(function(){
this.src=config.formatterHelpers.resolvePath(this.src,false);
return false;
});
} else
src=config.formatterHelpers.resolvePath(src,true);
}
img.src=src;
w.nextMatch = this.lookaheadRegExp.lastIndex;
}
}
config.formatterHelpers.imageSize={
tip: 'SHIFT-CLICK=show full size, CTRL-CLICK=restore initial size',
dragtip: 'DRAG=stretch/shrink, '
}
config.formatterHelpers.addStretchHandlers=function(e,stretchW,stretchH) {
e.title=((stretchW||stretchH)?this.imageSize.dragtip:'')+this.imageSize.tip;
e.statusMsg='width=%0, height=%1';
e.style.cursor='move';
e.originalW=e.style.width;
e.originalH=e.style.height;
e.minW=Math.max(e.offsetWidth/20,10);
e.minH=Math.max(e.offsetHeight/20,10);
e.stretchW=stretchW;
e.stretchH=stretchH;
e.onmousedown=function(ev) { var ev=ev||window.event;
this.sizing=true;
this.startX=!config.browser.isIE?ev.pageX:(ev.clientX+findScrollX());
this.startY=!config.browser.isIE?ev.pageY:(ev.clientY+findScrollY());
this.startW=this.offsetWidth;
this.startH=this.offsetHeight;
return false;
};
e.onmousemove=function(ev) { var ev=ev||window.event;
if (this.sizing) {
var s=this.style;
var currX=!config.browser.isIE?ev.pageX:(ev.clientX+findScrollX());
var currY=!config.browser.isIE?ev.pageY:(ev.clientY+findScrollY());
var newW=(currX-this.offsetLeft)/(this.startX-this.offsetLeft)*this.startW;
var newH=(currY-this.offsetTop )/(this.startY-this.offsetTop )*this.startH;
if (this.stretchW) s.width =Math.floor(Math.max(newW,this.minW))+'px';
if (this.stretchH) s.height=Math.floor(Math.max(newH,this.minH))+'px';
clearMessage(); displayMessage(this.statusMsg.format([s.width,s.height]));
}
return false;
};
e.onmouseup=function(ev) { var ev=ev||window.event;
if (ev.shiftKey) { this.style.width=this.style.height=''; }
if (ev.ctrlKey) { this.style.width=this.originalW; this.style.height=this.originalH; }
this.sizing=false;
clearMessage();
return false;
};
e.onmouseout=function(ev) { var ev=ev||window.event;
this.sizing=false;
clearMessage();
return false;
};
}
//}}}
!Links:
*[[VIS 2014 Schedule Browser|http://gramaz.io/vis/2014-schedule.html]] by [[Connor Gramazio|http://gramaz.io/]]
*[[Collaborative live notes (GoogleDoc)|https://docs.google.com/document/d/1IGSKD2gYE8_ay1z9jz2pFfmUeR7aVVe-d0mhDTnjnIA/mobilebasic?pli=1]]
*[[R. Kosara's blog|https://eagereyes.org/section/blog/2014]]
*[[S. Haroz's blog post on eval at InfoVis|http://steveharoz.com/blog/2014/guide-to-user-performance-evaluation-at-infovis-2014/]]
*[[S. Haroz's blog post on methods papers at InfoVis|http://steveharoz.com/blog/2014/infovis-2014-the-methods-papers/]]
!Techniques
!!~UpSet: Visualization of Intersecting Sets
Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, Hanspeter Pfister
[>img(100%, )[UpSet|https://pbs.twimg.com/media/B2ZMrlhCMAAE3dx.jpg]]
*Simpsons example: Venn diagrams of evil, blue hair, school, power plant, duff beer: doesn't scale
*area preserving Venn diagrams: difficult to estimate area sizes
*setviz.net
*horizon bar charts and aggregates, queries with must have, may have, and not have operators
*[[vcg.github.io/upset|http://vcg.github.io/upset]] live tool, upload your dataset
!!~OnSet: A visualization technique for large-scale binary set data
Ramik Sadana, Timothy Major, Alistair Dove, John Stasko
*QR code metaphor. 15x15 grid list, fixed sequence of elements.
*linking and brushing, overlap with hierarchical and/or queries and color coding of overlaps
*similarity of sets with bands of varying thickness
*domains: biological compounds, calendar data, finding availabilities of multiethnic people, US senate voting
*[[cc.gatech.edu/gvu/ii/setvis/|http://cc.gatech.edu/gvu/ii/setvis/]] Google Stasko onset
*q: linking and brushing and labelling of set elements? relies on spatial memory?
!!~DimpVis: Exploring Time-varying Information Visualizations by Direct Manipulation
Brittany Kondo, Christopher Collins
*''CPSC 547''
*gapminder example: animation, small multiples, individual items not easily tracked, global time slider not useful for local change
*object centric temporal navigation. Time hint path navigation for scatter plots, bar charts, pie charts, heatmaps q: overlap in heatmaps?
*evaluation on touch screen. q: what about mouse wheel?
*possible application to storytelling
*[[vialab.science.uoit.ca/dimpvis|http://t.co/necZ49GUoU]]
*q: what about multiple attributes? Comparison?
!!Axis Calibration for Improving Data Attribute Estimation in Star Coordinates Plots
Manuel ~Rubio-Sᮣhez, Alberto Sanchez
*conference slide templates. Math heavy. Cluttered, hard-to-read slides. staggered delivery. Tuning out.
*hybrid of statistical bi plots and star coordinates for estimating attribute values
*Eval? Strategies? Familiarity with star coordinates?
!!Domino: Extracting, Comparing, and Manipulating Subsets across Multiple Tabular Datasets
Samuel Gratzl, Nils Gehlenborg, Alexander Lex, Hanspeter Pfister, Marc Streit
[>img(100%, )[Domino|https://pbs.twimg.com/media/B2KEhhgIAAAPlFz.jpg]]
*''CPSC 547''
*[INFOVIS HONORABLE MENTION AWARD]
*blocks and subsets and relationships between them
*[[domino.caleydo.org|http://caleydo.org/projects/domino/]]
*demo video very fast
!Maps & Trees
!!Attribute Signatures: Dynamic Visual Summaries for Analyzing Multivariate Geographical Data
Cagatay Turkay, Aidan Slingsby, Helwig Hauser, Jo Wood, Jason Dykes
*''CPSC 547''
*small multiples linked view environment: census data example. Geographic linking and brushing
*paths and zooms on a map, attributes that vary in time
!!~Origin-Destination Flow Data Smoothing and Mapping
Diansheng Guo, Xi Zhu
*MAUP problem, hairball flow maps of migration, flow smoothing and normalization, flow thresholds
*spatialdatamining.org
*holds mic far away.
*experiments with synthetic data with injected flow clusters.
*shows retirees on east coast migrate to Florida, west coast to Arizona
!!Stenomaps: Shorthand for Shapes
Arthur van Goethem, Andreas Reimer, Bettina Speckmann, Jo Wood
[>img(100%, )[Stenomaps|https://pbs.twimg.com/media/B2KkgoPIQAIARYC.jpg]]
*''CPSC 547''
*trade off between area and boundary of shape to draw complex polygon (ie country) as single continuous curve
*what could we use this for? Labelling along lines, flow along lines, quantitative data with pictograms
*exact maps are perceived as exact data. Sketchy maps
*effectiveness? Recognizability? Task appropriateness? No user study yet
*are coasts more important than borders?
!!Nmap: A Novel Neighborhood Preservation Space-filling Algorithm
Felipe S. L. G. Duarte, Fabio Sikansi, Francisco M. Fatore, Samuel G. Fadel, Fernando V. Paulovich
*presented by FVP, Duarte a Msc student.
*proximity-similarity gestalt principle violated in treemaps. Spatial ordered treemaps
*less displacement of points, ~NMap, Neighborhood preservation, aspect ratio optimization
*evaluation vs. single dimension ordered treemaps, 2d ordered treemaps
*a good candidate to generate cartograms, text document collections, hierarchically
*MDS preserves distance-similarity metaphor
!!Tree Colors: Color Schemes for ~Tree-Structured Data
Martijn Tennekes, Edwin de Jong
*treemap R package, JS library forthcoming
*Shiny! Live demo
*reversals and permutations to aid in discrimination. Hue families for branches from origin
*also applied to treemap
*[[R package|http://cran.r-project.org/web/packages/treemap/]]
!Networks
!!Multivariate Network Exploration and Presentation: From Detail to Overview via Selections and Aggregations
Stef van den Elzen, Jarke J. van Wijk
*''CPSC 547''
*[INFOVIS BEST PAPER AWARD]
*Namredienhs: reverse Shneiderman mantra
[>img(100%, )[Namredienhs|https://pbs.twimg.com/media/B2K5ishIIAAbU90.jpg]]
*DOSA: Detail to Overview via Selections and Aggregations
*direct manipulation queries, scented widgets
*live demo: see supplemental material, video
!!~GLO-STIX: ~Graph-Level Operations for Specifying Techniques and Interactive eXploration
Charles D. Stolper, Minsuk Kahng, Zhiyuan Lin, Florian Foerster, Aakash Goel, John Stasko, Duen Horng Chau
*''CPSC 547''
*live demo graph vis design authoring tool
*graphs, semantic substrates, pivot graphs
*fw: interaction
*Q: best practices / recommendations for graph vis?
!!A Modular ~Degree-of-Interest Specification for the Visual Analysis of Large Dynamic Networks,
James Abello, Steffen Hadlak, Heidrun Schumann, ~Hans-J沧 Schulz
*~DoI functions for dynamic networks.
*linked views, histograms, dynamic graphs, stacked graph, animation
*Hoover over the namesakes (scenario)
!!~GraphDiaries: Animated Transitions and Temporal Navigation for Dynamic Networks
Benjamin Bach, Emmanuel Pietriga, ~Jean-Daniel Fekete
*[TVCG]
*brain connectivity example, migration patterns
*animation for each time step: memory requirement
*task taxonomy for dynamic networks based on spatiotemporal task taxonomy by Donna Perquet, 1994: what, where, when
*small multiples and animation, colour highlighting for added and removed nodes
*selection of nodes to highlight over time
*controlled user study
*read for Infovis group?
*q: TJ JK on rocking between states, task taxonomy dimensions
!!Visual Adjacency Lists for Dynamic Graphs
Marcel Hlawatsch, Michael Burch, Daniel Weiskopf
*''CPSC 547''
*[TVCG]
*large space-filling, sparse layout
*again, migration patterns
*adjacency matrix vs. lists
*hard to follow
!!How to Display Group Information on ~Node-Link Diagrams: an Evaluation
Radu Jianu, Adrian Rusu, Yifan Hu, Douglas Taggart
*[TVCG]
*many different tasks evaluated, four different node-link visualization designs
*accuracy, not time, the interest.
*eye tracking done as well, examination of visual clutter.
*gmap, bubble, node, lines
*end of day. Burnt out. Overlapping sets not handled.
*q: where did tasks come from? See paper. Group tasks made up. Task groupings between subjects?
!Interaction and authoring
!!Revisiting Bertin matrices: New Interactions for Crafting Tabular Visualizations
Charles Perin, Pierre Dragicevic, ~Jean-Daniel Fekete
*Bertin's other, less well-known book on graphics for information processing: Bertin foresaw the utility of interaction
*bertifier.com, live demo of tool, [[aviz.fr/bertifier|http://aviz.fr/bertifier]]
*Negation, reordering, sorting rows and columns
*changing the encoding of the rows and columns
*manual annotation and grouping
*using Bertifier to display results from experiment, such as Likert data
*q: line between exploration and presentation is fuzzy with Bertifier
!!iVisDesigner: Expressive Interactive Design of Information Visualizations
Donghao Ren, Tobias H欬erer, Xiaoru Yuan
*how does this differ from Lyra? Vector graphics used in iVisDesigner (not in Lyra?)
*live demo
*les miserables character graph
*brushing and linking different from Lyra, dragging with inverse mapping to update data
*RW: Tableau, ~SketchStory,
*cars dataset
*object oriented framework
*streaming dataset capabilities
*[[donghaoren.org/ivisdesigner|https://donghaoren.org/ivisdesigner/]]
*q: do you need to be a programmer to use it? Maybe? Can you export it? Yes.
!!Constructing Visual Representations: Investigating the Use of Tangible Tokens
Samuel Huron, Yvonne Jansen, Sheelagh Carpendale
*Bret Victor's talk on drawing dynamic visualizations
*DIS paper on constructive visualization
*used bertifier to display participant demographics
*SC spoke about this work at CANVAS.
*[[constructive.gforge.inria.fr|http://constructive.gforge.inria.fr/#!index.md]]
*token kit available
*q: could this approach be used in general Vis design?
!!~PanoramicData: Data Analysis through Pen & Touch
Emanuel Zgraggen, Robert Zeleznik, Steven Drucker
*Heer and Shneiderman taxonomy shoutout, support for tasks
*tasks separated into different views; frequent task switching
*RW: Lark, ~DataMeadow, ~GraphTrail, Dimension Projection Matriz/Tree (Yuan 2013)
*Browne et al 2011 data analysis on whiteboards
*derivable visualizations, exposing expressive data operations (Polaris), unbounded space, Boolean composition
*wine is sunlight held together by water - Steven Spurrier
*recorded demo with linking and tagging and saving tags of subsets (grad wines)
*joins and SQL translations of queries. FW: advanced statistical analysis, data cleaning
!!Munin: A ~Peer-to-Peer Middleware for Ubiquitous Analytics and Visualization Spaces, Sriram Karthik Badam, Eli Raymond Fisher, Niklas Elmqvist
*[TVCG]
*Marc Weiser quote on ubicomp.
*DC presentation overlap
*showing code. Too much ubicomp for Vis audience?
*~PolyZoom by Javed 2012
*media player? Too much chi for Vis audience?
*~PolyChrome at ITS 2014
*[[github.com/karthikbadam/munin|https://github.com/karthikbadam/Munin]]
! EDA
!!The Effects of Interactive Latency on Exploratory Visual Analysis
Zhicheng Liu, Jeffrey Heer
*talk by JH. Visa problems for ZL.
*immens from ~EuroVis 13, Nanocubes, progressive results, superconductor
*was immens worth it? Low latency systems
*+500ms latency at Google reduced number of searches by 20%, effect persists for days,
*what about latency and EDA?
*1s latency too long. Participants quit in pilots
*insight based methodology. Analytic events not insights. Observation, generalization, hypothesis, question, recall of existing knowledge, simulation, interface comments. Heer CHI 2008, Viegas HICSS 2007
*linear mixed-effects models
*many subjects did not perceive latency differences, deny that latency has effect on their interactions (in contrast to quantitative results)
*exposure to delay reduces later exploration in low latency condition (interaction effect)
*[[idl.washington.edu/papers/latency|http://idl.cs.washington.edu/papers/latency/]]
!!Visualizing Statistical Mix Effects and Simpson鳠Paradox
Zan Armstrong, Martin Wattenberg
[>img(100%, )[Simpson鳠Paradox|https://pbs.twimg.com/media/B2O6rfUCUAIORA6.jpg]]
*median wage overall increases, individual category median wage decreases
*mix effects can break many visualizations
*problems with treemaps
*ZA motivated work as financial analyst at Google
*displaying aggregate with individual points in Comet charts, able to spot outliers. Sometimes aggregate change is very small
*internal deployment at Google
*lots of open questions about understanding, non-time related data
*q: on angle discrepancy? Gestalt, flow similarity more important
*q on drill down, adding additional details
*q on direction of comet encoding.
!!Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Michael Correll, Michael Gleicher
[>img(100%, )[Don't use error bars|https://pbs.twimg.com/media/B2PAdOXCcAA269K.png]]
*''CPSC 547''
*don't use error bars. Use gradient or violin plots instead
*within-the-bar bias, containment bias
*design requirements for error bar replacement: consistent, symmetric, continuous
*meaning of error bars highly overloaded
*violin plot or gradient plot instead? Alpha channel a problem with the latter. Boosted alpha to correct such that values aren't near white from within the 95% CI
*people predict anger effect sizes with error bars. The all or nothing principle
*people have a hard time to interpret new encodings. Judgments of p value and effect size are in the right direction with new encodings
*[[graphics.cs.wisc.edu/Vis/errorbars/|http://graphics.cs.wisc.edu/Vis/ErrorBars/]] make your own
*Naomi Robbins Q on why not Cleveland dot plots. Not visually symmetric. Continuity important. Box plots are overloaded. ~M쬬er-Lyer illusion
*Q from statistician. Error bars show uncertainty. Why is symmetry important? Alternative error encodes such as gradient or violin plots are for inferential data from testing scenarios, not descriptive stats of distributions
*don't show gradient plots for distribution data
!!Four Experiments on the Perception of Bar Charts
Justin Talbot, Vidya Setlur, Anushka Anand
*only three experiments covered in talk. Graphical perception talk.
*separation effect. Attempting to explain results of Cleveland / ~McGill, Heer / Bostock
*separation, distractions, interaction effect with heights of bars
*experiment one: separation itself is important plus a small effect of tall distractors.
*open question: which aspect ratio should be used for bar charts?
*design implication: use a reference line for bar height comparison when bars are separatedn quai
*experiment two with stacked bars, distractors and dot position
*implication: favour aligned comparisons when possible. Grey out distractors or emphasize comparison bars
*open question about the role of distractors in stacked bar charts
*experiment one: comparing bar segments within the same bar and varying the number of distractors between them, distance between segments
*avoid comparing heights of adjacent stacked segments. Separation of segments with distractors diminished bias effect
*Q: Shneiderman on horizontal and vertical saccade differences. Perhaps make them horizontal bar charts and replicate experiment
*Q: what about bar thickness? Unknown.
*Q: Stacked vs grouped bars? Depends on task
!!Visual Parameter Space Analysis: A Conceptual Framework
Michael Sedlmair, Christoph Heinzl, Harald Piringer, Stefan Bruckner, Torsten M欬er
*motivated by design studies using input-output models
*how do these application domain problems transfer?
*three parts of framework: data flow model, analysis tasks, navigation strategies
*surrogate model instead of deriving from new output points in model (expensive)
*navigation strategies in addition to trial and error
*people follow the Overview first mantra based on their lit review
*analysis tasks: optimization, partitioning, fitting, outliers, uncertainty, sensitivity
*using the framework? Use it in interviews with analysts to understand and extend tasks
!Design and Persuasion
!!Moving beyond sequential design: Reflections on a rich multichannel approach to data visualization
Jo Wood, Roger Beecham, Jason Dykes
*design study approach / rethinking
*shoutout to the DSM, pitfalls
*bike docking domain problem
*trust and credibility is important. Task clarity is important. Trust, capability, efficacy
*storytelling uptake and media attention. Meanwhile, a much more analytical task was addressed for the London transport authority. Media attention and public engagement led to credibility
*feedback loop of credibility and analysis and public engagement. TEDX talk. Museum exhibit. Developing work for two audiences in parallel
*multiple design studies in parallel moving in different directions in the task clarity and information location space
*engineering serendipity and multi-channel design, parallel engagement
*Q: publication strategy? Multiple opportunities. Publish in domain literature.
*Q: how to replicate? Depends on what you mean by replication. What is it that engages people? Look for connections that resonate with different audiences.
!!An Algebraic Process for Visualization Design
Gordon Kindlmann, Carlos Scheidegger
[>img(100%, )[Algebraic vis|https://pbs.twimg.com/media/B2PpyPTCYAELQia.jpg]]
*INFOVIS HONORABLE MENTION AWARD
*[[algebraicvis.net|http://algebraicvis.net/]]
*principle of visual data correspondence. Congruence, effectiveness. Color map and scattershot examples, jumbles and misleaders
*unambiguous data depiction principle, confusers
*representation invariance principle, hallucinators
*[[slides|http://algebraicvis.net/2014/11/11/presentation_slides.html]]
!!Design Activity Framework for Visualization Design
Sean ~McKenna, Dominika Mazur, James Agutter, Miriah Meyer
*creative visualization redesign
*methods from design community
*love/breakup letters
*process timelines are messy. Embrace the messiness
!!Activity Sculptures: Exploring the Impact of Physical Visualizations on Running Activity
Simon Stusak, Aur鬩en Tabard, Franziska Sauka, Rohit Ashok Khot, Andreas Butz
*aesthetically pleasing sculptures. Reminds me of beyond the desktop interaction model from ~InfoVis 13
*curiosity and playfulness
*visualize running activity as a sculpture, necklace, robot, jar
*challenges include sustainability with 3d printing
!!The Persuasive Power of Data Visualization
Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite, Enrico Bertini
*''CPSC 547''
*video games and youth violence Likert scale question. Evidence, tabular data, graphical depiction.
*does Vis make argument persuasive?
*claims that no Vis literature assesses persuasive power of Vis, but lots of online four debate on persuasive infographics.
*what is persuasion? How do you study it? He do you create an experiment to study persuasion and Vis?
*interestingness and polarizing. How to define these?
*isn't polarization of topics highly geographically dependent?
*boomerang effect
*public understanding of science journal, business insider article on convincing nature of charts
!Evaluation
!!Comparative Eye Tracking Study on ~Node-Link Visualizations of Trajectories
Rudolf Netzel, Michael Burch, Daniel Weiskopf
*GPS data with ubiquitous tracking
*directed graphs with different encodings. Tapered links, standard arrows, tapered comets, equidistant comets
*link ordering and halo edge / node crossings, node encodings to represent node density
*tasks: path following, recognition of special links, estimation of cluster density
*tapered links performed well. Depth sorting positive result. Point splitting helpful.
*q: aren't tasks directly computable? Why is visualization needed? Low-level elementary tasks. What about higher-level tasks?
!!Node, ~Node-Link, and ~Node-Link-Group Diagrams: An Evaluation
Bahador Saket, Paolo Simonetto, Stephen Kobourov, Katy Borner
*XKCD references to spatialization of abstract information
*scatter plot, node link diagrams, and maps
*overlap with Jianu et al 2014 (see above)
*"I will explain you now"
*tasks from ~EuroVis 2014 paper
*all three encodings for node based tasks, node-link and maps for network based tasks, maps for group based tasks
*q: how do people do network tasks with node diagrams? Spatial similarity
*q: are tasks driven by different analytical needs?
!!The ~Not-so-Staggering Effect of Staggered Animated Transitions on Visual Tracking
Fanny Chevalier, Pierre Dragicevic, Steven Franconeri
*staggering animations common in many Vis tool kits
*[[fannychevalier.net/animations|http://fannychevalier.net/animations]] for replication
*confidence intervals and estimation and not significance testing ࠬa BELIV keynote
*shape of mark may have impact
*staggered animation is not staged animation
!!The Influence of Contour on Similarity Perception of Star Glyphs
Johannes Fuchs, Petra Isenberg, Anastasia Bezerianos, Fabian Fischer, Enrico Bertini
*''CPSC 547''
*whiskers glyph is a star glyph without a contour
*effect of contour closure. Whiskers glyph, star glyph, and sensitivity star glyph (contour only)
*between subjects factor of expertise, within subject factor of glyph, within subject factor of data density ( dimensionally of glyphs)
*effect of fill on shape.
*distractors for determining similarity in data space
*error bars
*no indication that subjects were looking at data. They were told to look for similar shapes
*whisker glyphs are best for high data density
*adding tick marks and grid lines / circular contours to whisker plots in third study. Implications for visual clutter
*what is expertise? ~PhDs in visualization or students working for them.
*ex-speer-iment
!!Order of Magnitude Markers: An Empirical Study on Large Magnitude Number Detection
Rita Borgo, Joel Dearden, Mark W. Jones
*horizon chart bar chart?
*dual scale charts and transformations (Isenberg). Broken axis charts and panel charts
*scale-stack bar charts (~EuroVis 13)
*~OOOMs are about as accurate as linear markers, faster
*~OOOMs are good for identifying outliers (in terms of accuracy, not at the expense of response time)
*ratio estimation task: OOMS good in terms of accuracy, RT did not increase
*trend analysis task: all representations similarly accurate and fast
*extended to stacked bar charts? After Talbot's talk. What about data for very similar ranges?
*q: what about thresholds or OOM? 99 and 101? Interesting case.
*q: binary error response? Systematic errors qualitatively defined?
!Applications
!!~LiveGantt: Interactively Visualizing a Large Manufacturing Schedule
Jaemin Jo, Jaeseok Huh, Jonghun Park, Bohyoung Kim, Jinwook Seo
*scalability of Gantt charts. Explorability. Reschedulability.
*reordering and task aggregation, highlighting
*RW: ~LifeFlow by Wonguphasawat and Shneiderman
*filtering and history. Juxtaposition and linking between Gantt view and performance view with linked focus line. Previews for moving tasks in Gantt chart
*[[hcil.snu.ac.kr/research/livegantt|http://hcil.snu.ac.kr/research/livegantt]]
!!~TenniVis: Visualization for Tennis Match Analysis
Tom Polk, Jing Yang, Yueqi Hu, Ye Zhao
*glyph based encoding for games. Fish grid extended at deuces.
*zooming out to display the entire game. Pie meter to show dominance by one player or another
*colour-blind safe?
*non-spatial encoding of tennis match. PW visualized location of player or ball.
*game win probability filter. Visual queries. Showing how a player might choke.
*watching tennis to show how players choke
*pilot study with tennis coaches at UNCC
*video linking to events.
*q: data entry? Manual on a mobile device after each point.
*q: assumptions of overview level vs. detail level. Why not different visual encoding at different zoom Levels?
!!Combing the Communication Hairball: Visualizing ~Large-Scale Parallel Execution Traces using Logical Time
Katherine Isaacs, ~Peer-Timo Bremer, Ilir Jusufi, Todd Gamblin, Abhinav Bhatele, Martin Schulz, Bernd Hamann
*existing Vis: Gantt-like depiction of processes over time. Doesn't scale. Hairball.
*structure-centric rather than physical time
*encoding physical time as colour over logical time. Eg. Lateness / delay
*clustering timelines together, hierarchical clustering, focus processes
*Ravel tool. Linked views with logical timelines, cluster timelines, physical timelines, lateness overview
*case studies show improved ordering of parallel processes, detecting strange delay patterns, detecting interaction effects between processes
!!~MovExp: A Versatile Visualization Tool for ~Human-Computer Interaction Studies with 3D Performance and Biomechanical Data
Gregorio Palmas, Myroslav Bachynskyi, Antti Oulasvirta, ~Hans-Peter Seidel, Tino Weinkauf
*DSM shoutout. Animated movement of human arm motion
*selections across linked views
*visual linking to body parts, selection of muscles to view corresponding movement patterns in 3d trajectories
*developed in two years, 2 deployments
*application to the design of movement patterns for public displays
!!~NeuroLines: A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity
Ali ~Al-Awami, Johanna Beyer, Hendrik Strobelt, Narayanan Kasthuri, Jeff W. Lichtman, Hanspeter Pfister, Markus Hadwiger
*''CPSC 547''
*[INFOVIS HONORABLE MENTION AWARD]
*scalability problems and visual clutter with existing 3d volumetric visualization approaches
*small multiples
*case study evaluation
*like Borkin's artery Vis transforms
*[[rhoana.org/neurolines/|http://www.rhoana.org/neurolines/]]
!Perception and Design
!!Learning Perceptual Kernels for Visualization Design
ǡay Demiralp, Michael Bernstein, Jeffrey Heer
*shape confusion matrix and clustering of visual variables
*what are they useful for? Palette design and automatic visualization. Visual embedding, a model for visualization: rank correlations
*shape, size, colour, their combinations
*five tasks, 600 subjects on ~MTurk,
*paper is more about methods than findings. Judgment task to arrange times in terms of similarity.
*Steven's power law explains the results for all but one task (spatial arrangement)
*distance matrixes for perceptual judgment tasks. Avoid manual spatial arrangement as a method.
!!Ranking Visualization of Correlation Using Weber鳠Law
Lane Harrison, Fumeng Yang, Steven Franconeri, Remco Chang
*(at VAST serendip talk)
*[[github.com/TuftsVALT/ranking-correlation|https://github.com/TuftsVALT/ranking-correlation]]
!!The Relation Between Visualization Size, Grouping, and User Performance
Connor Gramazio, Karen Schloss, David Laidlaw
*PW on aspect ratio and monitor size
*set size and mark size
*Haroz and Whitney on the pop out effect
*experiment 1: size in tabular Vis.
*75 conditions. 2400 trials. 36000 trials total
*effect size detracts from performance for large set sizes
*increased mark sizes improve performance, but with diminishing returns
*multiple linear regression
*experiment 2: scatter plots with grouping and random layouts
*similar trends for set size and mark size, but longer ~RTs
*non-designers can intuit performance based on survey results
*case study with cancer researchers
*gramaz.io
*q: increasing mark size and quadrants
*q: mark size increase leads to asking different task?
*q: direction of reading? No significant effect
!!A Principled Way of Assessing Visualization Literacy
Jeremy Boy, Ronald Rensink, Enrico Bertini, ~Jean-Daniel Fekete
*Vis for the masses
*what is ~VisLit? Analogy with text. Visualization, ~VisLit, data comprehension.
*item response theory
*congruent and incongruent questions.
!!Reinforcing Visual Grouping Cues to Communicate Complex Informational Structure
Juhee Bae, Benjamin Watson
*tuned out / burnt out
!Documents, Search, Images
!!Overview: The Design, Adoption, and Analysis of a Visual Document Mining Tool For Investigative Journalists
Matthew Brehmer, Stephen Ingram, Jonathan Stray, Tamara Munzner
!!How Hierarchical Topics Evolve in Large Text Corpora
Weiwei Cui, Shixia Liu, Zhuofeng Wu, Hao Wei
*''CPSC 547''
!!Exploring the Placement and Design of ~Word-Scale Visualizations
Pascal Goffin, Wesley Willett, ~Jean-Daniel Fekete, Petra Isenberg
!!Similarity Preserving ~Snippet-Based Visualization of Web Search Results
Erick ~Gomez-Nieto, Frizzi San Roman, Paulo Pagliosa, Wallace Casaca, Elias S. Helou, Maria Cristina Ferreira de Oliveira, Luis Gustavo Nonato
*[TVCG]
!!Effects of Presentation Mode and Pace Control on Performance in Image Classification
Paul van der Corput, Jarke J. van Wijk
!References
<<bibliography InfoVis-2014-bibtex showAll>>
*~InfoVis website research
**[[Berkeley Visualization Lab |http://vis.berkeley.edu/]] - Agrawala
***+recent pubs w/ thumbs front and centre, simple design, local search, recent news headlines
***ଥ not on front page, no brief bios
**[[StanfordVis Group|http://vis.stanford.edu/]] - Heer / Hanrahan
***+recent pubs w/ thumbs front and centre, highlighted projects, simple design, recent news headlines
***ଥ not on front page, sidebar headers hard to discern, no brief bios
**[[Brown Visualization Research LAb|http://vis.cs.brown.edu/]] - Laidlaw
***볠ancient, mouse-over navigation menus, lots of scrolling
***�ous headers: results, organization
**[[UNC Charlotte Visualization Center|http://viscenter.uncc.edu/]] - Ribarsky, Kosara
***+highlight reel, local search, decent people page w/ thumbs and brief bio/interests, recent news headlines
***宴 pubs not on front page, no pubs thumbs, lots of scrolling
**[[Virgina Tech InfoVis Lab|http://infovis.cs.vt.edu/]] - North
***Წy a ~WordPress job, no recent pubs, people on front page
***ଥ page inconsistent (only some thumbs), no bios / interests
***㠨ave no abstract / thumbs
**[[Calgary InnoVis|http://innovis.cpsc.ucalgary.ca/]] - Carpendale
***魡l splash page, overly simple design, people not on front page, no bios/interests/thumbs, no thumbs / abstracts for pubs
**[[INRIA Aviz|http://www.aviz.fr/]] - Fekete
***९ple on front page
***+recent news, pubs have thumbs
***ଥ thumbs silly/ambiguous (just eyes), no interests/brief bios
**[[Georgia Tech Information Interfaces|http://www.cc.gatech.edu/gvu/ii/]] - Stasko
***+people thumbs, project thumbs, projects, news on front page
***+people page lists affiliated projects
***�navigation menu, Dilbert comic, lots of white space, scrolling
**[[Fernanda B. Vi쨴tp://fernandaviegas.com/]]
***+simple, visual, theme-driven
***䲡st, too much colour
**[[SFU Gr贴p://gruvi.cs.sfu.ca/]] - M⊪**+recent news w/ images
***९ple, recent pubs on home page, people page is flat list with tiny thumbs, no brief bios/interests
***㠨ave no thumbs/abstracts
**[[UVic VisID|http://visid.cs.uvic.ca/index.html]] - Tory
***+images of people on home page, es
***㠯n people page too long, inconsistent
***ൢs local, on Tory's site, no thumbs/abstracts
***塲ch projects a flat list, no browsing by theme
**[[UNH Data Visualization Research Lab|http://ccom.unh.edu/vislab/]] - Ware
***+research project thumbs on home page
***ଥ not on home page
***ଥ page has thumbs, no brief bios / interests
***㠬isted in research areas (not all areas have pubs)
**[[Harvard GVI|http://gvi.seas.harvard.edu/pfister]] - Pfister
***+recent pubs w/ thumbs on home page, people w/ thumbs and emails on home page
***୵ch scrolling
***⩯s / interests / affiliated projects
**[[City U. London giCentre|http://www.soi.city.ac.uk/organisation/is/research/giCentre/index.html]] - Dykes / Wood
***+recent projects/news w/ thumbs, brief abstracts on home page, ९ple on home page
***+people page has thumbs, interests, affiliations
***㩮g on home page: too cluttered
**[[StᲴ|http://www.visus.uni-stuttgart.de/institut.html]] - Ertl, Weiskopf, Fuchs
***+thumb of recent highlighted project, thumbs of ~PIs on home page, news and events
***ୡny menus (people hard to find); people an A-Z list, not by affiliation, status, project
*Attendees: TM, MB, MS, JD, SI
*Review / critique of textbook ch. 8
**Attribute reduction methods
***Synthetic dimensions - dimensional aggregation a form of dimension reduction?
***~DimStiller paper (Ingram et. al. (2010))
*Next week (Oct 13):
**Discussion of <<cite Lam2011 bibliography:Bibliography>>
*Attendees: TM, MB, MS, JD, SI, MM
*Discussion of <<cite Lam2011 bibliography:Bibliography>>
**Comments added to literature review in [[Information Visualization Evaluation: Meta-Analysis]]
*Next week: MS pre-paper talk (2pm, followed by DLS)
{{{
@article{Bach2014,
author = {Bach, B. and Pietriga, E. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {740--754},
title = {{GraphDiaries: Animated transitions and temporal navigation for dynamic networks}},
volume = {20},
year = {2014}
}
@article{Al-Awami2014,
author = {Al-Awami, A. K. and Beyer, J. and Strobelt, H. and Kasthuri, N. and Lichtman, J. K.. and Pfister, H. and Hadwiger, Markus},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2369--2378},
title = {{NeuroLines: A subway map metaphor for visualizing nanoscale neuronal connectivity}},
volume = {20},
year = {2014}
}
@article{Armstrong2014a,
author = {Armstrong, Z. and Wattenberg, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2132--2141},
title = {{Visualizing statistical mix effects and simpson's paradox}},
volume = {20},
year = {2014}
}
@article{Bae2014,
author = {Bae, J. and Watson, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1973--1982},
title = {{Reinforcing visual grouping cues to communicate complex informational structure}},
volume = {20},
year = {2014}
}
@article{Borgo2014,
author = {Borgo, R. and Dearden, J. and Jones, M. W.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Order of magnitude markers: An empirical study on large magnitude number detection}},
volume = {20},
year = {2014}
}
@article{Boy2014a,
author = {Boy, J. and Rensink, R. A. and Bertini, E. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{A principled way of assessing visualization literacy}},
volume = {20},
year = {2014}
}
@article{Brehmer2014c,
author = {Brehmer, M. and Ingram, S. and Stray, J. and Munzner, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2271--2280},
title = {{Overview: The design, adoption, and analysis of a visual document mining tool for investigative journalists}},
volume = {20},
year = {2014}
}
@article{Chevalier2014,
author = {Chevalier, F. and Dragicevic, P. and Franconeri, S.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2241--2250},
title = {{The not-so-staggering effect of staggered animated transitions on visual tracking}},
volume = {20},
year = {2014}
}
@article{Correll2014b,
author = {Correll, M. and Gleicher, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2142--2151},
title = {{Error bars considered harmful: Exploring alternate encodings for mean and error}},
volume = {20},
year = {2014}
}
@article{Cui2014a,
author = {Cui, W. and Liu, S. and Wu, Z. and Wei, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2281--2290},
title = {{How hierarchical topics evolve in large text corpora}},
volume = {20},
year = {2014}
}
@article{Demiralp2014,
author = {Demiralp, C. and Bernstein, M. S. and Heer, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1933--1942},
title = {{Learning perceptual kernels for visualization design}},
volume = {20},
year = {2014}
}
@article{Duarte2014a,
author = {Duarte, F. S. L. G. and Sikansi, F. and Fatore, F. M. and Fadel, S. G. and Paulovich, F. V.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2063--2071},
title = {{Nmap: A novel neighborhood preservation space-filling algorithm}},
volume = {20},
year = {2014}
}
@article{Fuchs2014,
author = {Fuchs, J. and Isenberg, P. and Bezerianos, A. and Fischer, F. and Bertini, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2251--2260},
title = {{The influence of contour on similarity perception of star glyphs data lines only ( D ) data lines + contour ( D + C ) contour only ( C )}},
volume = {20},
year = {2014}
}
@article{Goffin2014,
author = {Goffin, P. and Willett, W. and Fekete, J.-D. and Isenberg, P.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Exploring the placement and design of word-scale visualizations}},
volume = {20},
year = {2014}
}
@article{Gramazio2014,
author = {Gramazio, C. C. and Schloss, K. B. and Laidlaw, D. H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1953--1962},
title = {{The relation between visualization size , grouping , and user performance}},
volume = {20},
year = {2014}
}
@article{Gratzl2014a,
author = {Gratzl, S. and Gehlenborg, N. and Lex, A. and Pfister, H. and Streit, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2023--2032},
title = {{Domino: Extracting, comparing, and manipulating subsets across multiple tabular datasets}},
volume = {20},
year = {2014}
}
@article{Guo2014,
author = {Guo, D. and Zhu, X.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2043--2052},
title = {{Origin-destination flow data smoothing and mapping}},
volume = {20},
year = {2014}
}
@article{Harrison2014,
author = {Harrison, L. and Yang, F. and Franconeri, S. and Chang, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1943--1952},
title = {{Ranking visualizations of correlation using Weber's law}},
volume = {20},
year = {2014}
}
@article{Huron2014c,
author = {Huron, S. and Jansen, Y. and Carpendale, S.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2102--2111},
title = {{Constructing visual representations: Investigating the use of tangible tokens}},
volume = {20},
year = {2014}
}
@article{Isaacs2014a,
author = {Isaacs, K. E. and Bremer, P.-T. and Jusufi, I. and Gamblin, T. and Bhatele, A. and Schulz, M. and Hamann, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2349--2358},
title = {{Combing the communication hairball: Visualizing parallel execution traces using logical time}},
volume = {20},
year = {2014}
}
@article{Jo2014,
author = {Jo, Jaemin and Huh, Jaeseok and Park, Jonghun and Kim, Bohyoung and Seo, Jinwook},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2329--2338},
title = {{LiveGantt: Interactively Visualizing a Large Manufacturing Schedule}},
volume = {20},
year = {2014}
}
@article{Kindlmann2014b,
author = {Kindlmann, G. and Scheidegger, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2181--2190},
title = {{An algebraic process for visualization design}},
volume = {20},
year = {2014}
}
@article{Kondo2014,
author = {Kondo, B. and Collins, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2003--2012},
title = {{DimpVis: Exploring time-varying information visualizations by direct manipulation}},
volume = {20},
year = {2014}
}
@article{Lex2014,
author = {Lex, A. and Gehlenborg, N. and Strobelt, H. and Vuillemot, R. and Pfister, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1983--1992},
title = {{UpSet: Visualization of intersecting sets}},
volume = {20},
year = {2014}
}
@article{Liu2014,
author = {Liu, Z. and Heer, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2122--2131},
title = {{The effects of interactive latency on exploratory visual analysis}},
volume = {20},
year = {2014}
}
@article{McKenna2014,
author = {McKenna, S. and Mazur, D. and Agutter, J. and Meyer, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2191--2200},
title = {{Design activity framework for visualization design}},
volume = {20},
year = {2014}
}
@article{Netzel2014,
author = {Netzel, R. and Burch, M. and Weiskopf, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Comparative eye tracking study on node-link visualizations of trajectories}},
volume = {20},
year = {2014}
}
@article{Palmas2014,
author = {Palmas, G. and Bachynskyi, M. and Oulasvirta, A. and Seidel, H.-P. and Weinkauf, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2359--2368},
title = {{MovExp: A versatile visualization tool for human-computer interaction studies with 3D performance and biomechanical data}},
volume = {20},
year = {2014}
}
@article{Pandey2014a,
author = {Pandey, A. V. and Manivannan, A. and Nov, O. and Satterthwaite, M. and Bertini, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2211--2220},
title = {{The persuasive power of data visualization}},
volume = {20},
year = {2014}
}
@article{Perin2014b,
author = {Perin, C. and Dragicevic, P. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2082--2091},
title = {{Revisiting Bertin matrices: New interactions for crafting tabular visualizations}},
volume = {20},
year = {2014}
}
@article{Polk2014,
author = {Polk, T. and Yang, J. and Hu, Y. and Zhao, Y.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2339--2348},
title = {{TenniVis: Visualization for tennis match analysis}},
volume = {20},
year = {2014}
}
@article{Ren2014,
author = {Ren, D. and Hollerer, T. and Yuan, X.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2092--2101},
title = {{iVisDesigner: Expressive interactive design of information visualizations}},
volume = {20},
year = {2014}
}
@article{Rubio-Sanchez2014,
author = {Rubio-Sanchez, M. and Sanchez, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2013--2022},
title = {{Axis calibration for improving data attribute estimation in star coordinates plots}},
volume = {20},
year = {2014}
}
@article{Sadana2014,
author = {Sadana, R. and Major, T. and Dove, A. and Stasko, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1993--2002},
title = {{OnSet: A visualization technique for large-scale binary set data}},
volume = {20},
year = {2014}
}
@article{Saket2014c,
author = {Saket, B. and Simonetto, P. and Kobourov, S. and Borner, K.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2231--2240},
title = {{Node, node-link, and node-link-group diagrams: An evaluation}},
volume = {20},
year = {2014}
}
@article{Sedlmair2014a,
author = {Sedlmair, M. and Heinzl, C. and Bruckner, S. and Piringer, H. and Moller, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2161--2170},
title = {{Visual parameter space analysis: A conceptual framework}},
volume = {20},
year = {2014}
}
@article{Stolper2014b,
author = {Stolper, C. D. and Kahng, M. and Lin, Z. and Foerster, F. and Goel, A. and Stasko, J. and Chau, Duen Horng},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2320--2328},
title = {{GLO-STIX: Graph-level operations for specifying techniques and interactive eXploration}},
volume = {20},
year = {2014}
}
@article{Stusak2014,
author = {Stusak, S. and Tabard, A. and Sauka, F. and Khot, R. A. and Butz, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2201--2210},
title = {{Activity sculptures: exploring the impact of physical visualizations on running activity}},
volume = {20},
year = {2014}
}
@article{Talbot2014a,
author = {Talbot, J. and Setlur, V. and Anand, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2152--2160},
title = {{Four Experiments on the Perception of Bar Charts}},
volume = {20},
year = {2014}
}
@article{Tennekes2014,
author = {Tennekes, M. and de Jonge, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2072--2081},
title = {{Tree colors: Color schemes for tree-structured data}},
volume = {20},
year = {2014}
}
@article{Turkay2014,
author = {Turkay, C. and Slingsby, A. and Hauser, H. and Wood, J. and Dykes, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2033--2042},
title = {{Attribute signatures: Dynamic visual summaries for analyzing multivariate geographical data}},
volume = {20},
year = {2014}
}
@article{VandenElzen2014a,
author = {van den Elzen, S. and van Wijk, J. J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2310--2319},
title = {{Multivariate network exploration and presentation: From detail to overview via selections and aggregations}},
volume = {20},
year = {2014}
}
@article{VanderCorput2014,
author = {van der Corput, P. and van Wijk, J. J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2301--2309},
title = {{Effects of presentation mode and pace control on performance in image classification}},
volume = {20},
year = {2014}
}
@article{VanGoethem2014,
author = {van Goethem, A. and Reimer, A. and Speckmann, B. and Wood, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2053--2062},
title = {{Stenomaps: Shorthand for shapes}},
volume = {20},
year = {2014}
}
@article{Wood2014,
author = {Wood, J. and Beecham, R. and Dykes, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2171--2180},
title = {{Moving beyond sequential design: Reflections on a rich multi-channel approach to data visualization}},
volume = {20},
year = {2014}
}
@article{Zgraggen2014,
author = {Zgraggen, E. and Zeleznik, R. and Drucker, S. M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2112--2121},
title = {{PanoramicData: Data analysis through pen \& touch}},
volume = {20},
year = {2014}
}
}}}
!![Pirolli2005] - Sensemaking
[>img(40%, )[Pirolli's sensemaking loop for intelligence analysis, derived from cognitive task analysis|http://dydan.rutgers.edu/PDDALab/dev/images/flow.png]]
<<cite Pirolli2005 bibliography:Bibliography>> conducted a [[Cognitive Task Analysis]] and the [[Think Aloud Protocol]] to better understand the information foraging and sensemaking task-flow of intelligence analysts, providing a descriptive view of the process for the purpose of designing future technology.
At a high-level, the sensemaking process as described here is not at all different from the processes expert meteorologists engage in described by <<cite Trafton2000>>: getting a high-level overview, building a qualitative mental model, refining and adjusting the model, and extracting quantitative and qualitative information from that model in the form of decisions or a written brief (i.e. a weather forecast, flight plan). The figure at right captures these smaller sub-processes. <<cite Pirolli2005>> does however break down the tasks at a finer level, distinguishing between top-down and bottom-up processes, invoked in a "opportunistic mix".
The [[Cognitive Task Analysis]] results suggest several leverage points in the ''information foraging'' process in which technology may be able to improve efficiency or quality of the results. Each of these points are associated with costs, and therefore can be rephrased as heuristics when designing/evaluating tools to support the process:
*determine the exploration-enrichment-exploitation tradeoff - time and space allotted to monitoring, narrowing, and reading/analyzing. [[Focus+Context]] techniques serve as a compromise in this signal/noise tradeoff, one in which the costs of analyzing too many documents in detail (false positives) may be worth less than missing relevant documents (false negatives) - this cost differential is domain specific
*facilitate scanning, recognizing, selecting items for further attention - highlighting (i.e. pre-attentive codings)
*allow shifting attentional control - allowing bottom-up vs. top-down processing
*facilitate follow-up searches
In the ''sensemaking loop'', the [[Cognitive Task Analysis]] results suggest additional leverage points (focused around generating and managing hypotheses, reasoning and decision making), again rephrased as design/evaluation heuristics:
*use external working memory for analysts to manage evidence and hypotheses
*support adequate comparison of alternative hypotheses
*provide clear confirmation or dis-confirmation of hypotheses
!!!!Comments & Questions
*While the focus of this article is on intelligence analysts, I expect the processes depicted may generalize to a number of research fields involving domain expertise, schematic knowledge structures, pattern recognition and anomaly detection (the latter triggering the sensemaking loop).
*Missing references / references not numbered: where is Bodnar (2003)?
*Were these later rephrased as design/evaluation heuristics? Applied to other domains?
*Apparently there is another related paper regarding intelligence analysis in greater detail, or with a different focus - however web searches all point to this paper.
!![Pirolli2009] - ch. 1: Information Foraging Theory
<<<
E.O. Wilson, //Consilence//: "We are drowning in information, while starving for wisdom."
<<<
<<<
Kurt Vonnegut, //Breakfast of Champions//: "New knowledge is the most valuable commodity on earth. The more truth we have to work with, the richer we become."
<<<
In the introductory chapter of //Information Foraging Theory// (<<cite Pirolli2009>>), the author describes the basis of the theory, and discusses what will be addressed in the remainder of the book. The scope is broader than that of his earlier work involving intelligence analysis (Pirolli (1999), <<cite Pirolli2005>>), which focused on expert performance on well-defined tasks with application programs. This work aims to generally describe information foraging behaviour of any rational agents, with a heavy emphasis on information on the web. This is in response to the immense amount of data constantly being generated and added to the web, the many information retrieval methods and tools at our disposal, including external memory tools (i.e. [[Memory Extender]], [[Latent Semantic Analysis]]), and cognitive modelling to describe agent behaviour, such as Anderson's [[ACT-R|ACT]], or the current model used by <<cite Pirolli2009>>: [[ACT-Scent|ACT]] (pronounced ''accent'').
[>img(40%, )[Pirolli's ACT-Scent architecture / SNIF-ACT |http://www.websiteoptimization.com/speed/tweak/information-foraging/snif-act.png]]
The theory is based on optimal foraging theory, an established biological and anthropological theory for describing food foraging among animals. It involves a comparison between the average rate of gain (valuable information or food) in an environment of (food/information) patches against the actual gain curve over time. Whenever the slope of the tangent of the gain curve becomes less than the average rate of gain, the rational decision is to move to another patch. This is an optimization function seeking the maximum value divided by the effort expended. In the context of [[Human-Information Interaction]] (HII), it is the maximum (expected value of information gained / cost of interaction). HII is information-centric rather than computer centric (HCI).
The approach taken by the author is that of [[Rational Analysis]], which assumes ecological rationality, that evolving, behaving systems are rational, well-designed, and fulfill functions in context (a.k.a. [[Methodological Adaptationism|Rational Analysis]]). The human is not the agent of rational choice, but rather environmental forces shape rational choice. Human agents must make decisions and solve both well-defined and ill-defined problems under uncertainty (a bounded optimization / [[Satisficing]] problem). The author distinguished between the task environment and the information environment, as both will shape the rational choices needed. Information foraging can be described at the rational level (what environmental problem is being solved? why is the solution a good one?), the knowledge level (what does the system know?), the cognitive level (how does the system do it? functions described mechanistically by [[ACT-R|ACT]], other models such as SOAR, [[GOMS]], [[MHP]]), or at the biological level (how does the system physically do it?). The time scales of these levels of analysis ranges from days/weeks/months (rational) to fractions of a second (biological).
!!!!Comments & Questions
*Is HCI not user-centric? rather than computer-centric?
*A bit of a slog to read. Going to skip to ch. 9 next (design heuristics)
*Not much of an emphasis on expert behaviour / expert information foraging. Assumption of rationality for humans and optimal behaviour may vary considerably. The information environment can force rational decisions?
*difference between the task environment and the information environment = difference between visual analysis task and dataset?
*swamped with terminology: architecture, model, framework, theory...
!![Thomas2005] ch.2: The Science of Analytical Reasoning
In the second chapter of PNNL's //Illuminating the Path// research agenda, <<cite Thomas2005>> break down analytical reasoning, encompassing information foraging and [[Sense-making]]. It also encompasses a broader definition of the iterative and collaborative analytical process in general, including an overview of the types of analytical tasks (assessments, forecasts, development of opinions), and artifacts involved. [[Reasoning Artifacts]], [[Analytical Discourse]], and [[Sense-making]] are discussed with regards to their role in the analysis process, what is the state of the art, and what technology needs are recommended. The low-level roles of perception and cognition and the analytical processes that depend on them are also discussed. The chapter's final section pertains to collaborative analytic processes, reviewing work from the CSCW field.
The analytical process as defined here is similar/conforms to <<cite Pirolli2005>>'s cycles: determining how to address the issue being posed, what resources to use, planning to allocate time and resources to meet deadlines, gathering information and incorporating prior knowledge ([[Reasoning Artifacts]], create abstractions based on these artifacts, generating multiple explanations (alternate hypotheses) based on convergent and divergent thinking, forms judgments based on evidence gathered, checks assumptions and biases, and considers alternatives. Finally, the analyst presents or reports their judgments, recommends action.
Visual analytics and information visualization amplifies cognition, playing an important role in [[Analytical Discourse]] and [[Sense-making]], serving several functions:
*''increasing resources'': high-bandwidth displays, allowing parallel perception (unlike text), graphical rather than symbolic processing (offloading cognition to perception), expanding working memory, expanding information storage
*''reducing search time'': locality, high density, spatially-indexed
*''enhancing pattern recognition'': recognition instead of recall, abstraction and aggregation, visual schemata for organization, enhance patterns at all levels: values, relationships, trends
*''allowing perceptual inferences'': obvious problems / entities are apparent due to many pre-attentive perceptual processing, graphical computations possible
*''support interactivity''
The authors specify the needs of future technology to support analysis, as well as the needs of the research and development process that leads to this technology. Visually-based methods must be developed for the entire analytic process, from the gathering of artifacts to the weighing of alternative hypotheses. The need for a wider range of evaluation techniques is also among these recommendations: task analysis, field studies must complement lab studies and back-to-back testing. Low-level basic research must also be conducted to better understand the interaction between perception, cognition, attention, and visual analytics tools. In parallel to the above, call for the research and development of mixed initiative systems for assisting the human user in their analysis. Further development of collaborative tools is also called for.
!!!!Comments & Questions
*Draws from many fields of research to create an outline for future R&D (Cognitive Science, Business Intelligence, Perceptual Psychology, CSCW).
*Once you read a single subsection of this book, you realize there's a clear formula to each section: here's what's known with regards to subject x / here's what's been done (or what's the state of the art), and what are the recommendations for improving this process or better-integrating the process into further R&D.
*Definition heavy, many terms used to describe the analysis process and what it encompasses.
*There's an obvious lean towards applications in the intelligence and homeland security sector, as this is their mandate. Therefore it places emphasis on uncertainty and decision-making under time constraints and with various risks. Exploratory analysis and/or creative problem solving without strict consideration for time costs and risks is not a focus of the PNNL's project.
**This is increasingly obvious when discussing constraints of strict time pressures to come up with decisions under uncertainty, and the mention of deception in the data
*Taken together, descriptions of characteristics of how ~InfoVis amplifies cognition can be used as design/evaluation heuristics, albeit at a level without sensitivity to high-level or domain tasks
!![Yi2008] - Characterizing Insight
<<cite Yi2008>>'s BELIV '08 workshop paper attempts to answer questions relating to how and when insight is gained, rather than adding to the pile of previous work which attempted to explain what //insight// is and how it is characterized - which remains poorly understood (<<cite Mayr2010>> refers to this as a "//black box//"). It is a meta-review, not of evaluation methods or of qualitative research, but of gathered instances in previous work where insight was gained and via what means. It therefore fits into the notion of sensemaking and information foraging as insight is considered to be the central element of <<cite Pirolli2005>>'s sensemaking and information foraging model: //information > scheme > ''insight'' > product//. They posit that insight is not the product (as in <<cite Pirolli2005>>'s model), but a midpoint for more cycling and iterations back and forward towards a product (sensemaking is often retrospective).
They surveyed quotes from 4 books, 2 chapters, 34 papers, and determined that insight is gained during several overlapping but distinct processes: when providing an overview (big picture), when adjusting the level of abstraction (i.e. grouping / filtering), when detecting patterns, and when the data matches one's mental model. They also accounted for barriers to gaining insight: usability problems, poor motivation, lack of background/domain knowledge or training
The authors plan to develop insight-based heuristics for evaluation and design, as identifying how and when insight is gained is not sufficient for these tasks. They also plan to investigate how insight is gained with information visualization tools compared to other means of gaining insight.
!!!!Comments & Questions
*An //insight// on //insight// paper? (a few funny sentences such as this appear throughout)
*A "When and how? - not What?" meta-review similar to <<cite Lam2011>> re: evaluation scenarios - "when and how to choose an evaluation method", not what methods can I choose from?
**"When and how does insight happen?", rather than "what is insight?"
*Their section on barriers to insight does support the notion that evaluation should be carried out with real users in real situations with real data that they feel motivated about.
!![Andre2009] - On Serendipitous Discovery
<<cite Andre2009>>'s Creativity and Cognition conference paper reviews the definitions of serendipity, historical accounts of serendipitous insight and discovery, and how it has been interpreted in the Computing Sciences. It offers a meta-review of systems built to support serendipity, however the paper argues that these systems only account for one part of serendipity: the chance discovery of something unexpected, or something sought after in an unexpected location (the cause). Many systems do not account for the second aspect of serendipity (the effect), the //sagacity// or insight to acknowledge an unexpected connection with earlier knowledge and expertise, and the will to act upon these connections, by reinforcing an existing problem or solution, rejecting or confirming ideas, or starting a new research direction. There exists many existing systems that filter or suggest potentially relevant or interesting content to a user, akin to a recommender system (albeit recommender systems often rely popular content, as opposed to personalized content). These systems tend to be peripheral/background applications and occurrences of serendipitous discovery are rare, however users tend to be delighted when this happens. More often these systems are distracting or overwhelming.
The paper concludes with high-level design recommendations for providing opportunities to support this second aspect of serendipity, that ability to act on unexpected but useful information by drawing connections through analogy, inversion, or successful error. Systems would therefore have to prime users with familiar content in order to draw connections with new unexpected content, by exploiting a shared metalanguage or semantics across disciplines. Such a shared metalanguage would allow for serendipity-hunting agents to be built. Coupled with personalized content presentation (background/peripheral reporting, i.e. feed-readers) and a detailed understanding of a user's expertise (via life logging, heterogeneous sources of user information), this pairing could facilitate the second aspect of serendipitous insight. Integrating retroactive answering of search queries, novel presentation of search query results, and activities involving creativity, play, and aleatoricism may also promote better chance encounters (the first step of serendipity).
The authors also remark that the study of serendipity and insight is difficult due to their inherent rarity, particularly in a controlled setting. In naturalistic interview, diary, or search log studies, it is often difficult to identify specific instances of serendipitous discovery and insight. Nevertheless, they acknowledge the value in multiple approaches to studying this complex and rare phenomena across domains.
!!!!Comments & Questions
*The intro/motivation goes at great length to separate the two defining aspects of serendipity, however the structure of the rest of the paper dances back and forth between the two, leaving me questioning as to how much of a contribution they have made in terms of design recommendations for the second aspect of serendipity-supporting systems
*Design recommendations are dangerously hand-wavy, treading into Malcolm Gladwell territory (he is cited in the paper), history of science lessons
*Shared metalanguage across disciplines - I would need to familiarize myself w/ computational linguistics / semantic web research to see how far away we are from achieving this dream
*Differences between recommender systems and serendipity agents is still hazy despite their efforts to disentangle the two
*Designing personalized systems is the key to insight?
*Still a large reliance on the user to "have a prepared / open / questioning mind". Motivation and engagement will interact with the amount of priming needed to make serendipitous connections
*Glad they acknowledged the difficulty of measuring/quantifying insight and serendipitous discovery, even in naturalistic studies
*Hoping they would have expanded on the role of creativity and play in serendipitous discovery, perhaps with historical examples.
*Privacy concerns and resistance to "life logging", information-seeking tasks still distributed across many (potentially) unconnected devices and platforms, some of which not digital.
!![Klahr1999] - ~Meta-Review of Scientific Discovery
<<cite Klahr1999>>'s meta-review examines scientific discovery from four complimentary approaches: historical accounts of scientific discovery, laboratory experiments (both exploratory and controlled or hypothesis-driven), observational studies of scientists in context, and computational modelling. Aside from the intrinsic value of a psychological theory of the scientific discovery process, an understanding of this process could lend itself to the design of computer programs and instrumentation for aiding scientists in their endeavors. They also posit that expert systems development research are relevant to the theory of scientific discovery and vice-versa.
The journal article covers a lot of ground, citing related work and historical examples throughout, and describes the scientific discovery process as problem solving, not fundamentally unlike all forms of human problem solving behaviour. The distinction between normal everyday problem solving and scientific problem solving is the combination of strong and weak methods, the former incorporating a rich amount of domain expertise, methodology, and background, while the latter is domain-independent, incorporating trial and error, hill climbing, means-ends analysis, planning, and analogy - the latter of which being a bridge between strong and weak methods. With expertise, problems within one's domains are more likely to be well-defined than ill-defined, and reliance upon heuristic search-based weak methods more likely than trial-and-error.
They describe each means of studying scientific discovery with regards to several evaluative criteria: face validity, construct validity, temporal resolution, likelihood of discovering new phenomena, rigour and precision of measurements, control and factorability of independent variables, external validity, and social and motivational factors. While any individual means of studying the scientific discovery process cannot fulfill all of these criteria, they can be complimentary to each other. The authors present several case studies in which a compliment of methods is used to study the discovery process. For example, important scientific discoveries by Planck, Krebs, Balmer, and Faraday are studied first by an analysis of published research, lab diaries and autobiographical accounts, then by simulating the data, hypotheses, and independent variables in exploratory lab studies, and finally using computational models given the same initial starting point.
The paper also discusses the role of surprise and distinguishes between scientific investigations that are theory or hypothesis driven, and those that are driven by observation of an unexpected or surprising phenomena. Studying how computational models react to surprising phenomena during an investigation and how they change course sheds light on the behaviour scientists undertake under similar circumstances.
The paper includes a discussion of the role of analogical reasoning for formulating initial hypotheses, and the notion of multiple search spaces, reproduced from an earlier Klahr paper from 1988. They are working towards a general theory of scientific discovery, a special case of general problem solving, one that involves parallel search of a hypothesis space, an experiment space, and representation space (abstractions, visual representations, notation), and a strategy/instrumentation space.
They conclude with the reassuring message that domain experts can be studied without domain knowledge, at the level of weak methods, problem solving and phenomena recognition processes. There is acknowledgement of the relation between problem solving and creativity, and the observation of the child-scientist: the level of creative problem-solving exhibited by children is alluring and creative, suggestive of the generality of weak-methods as-yet uninformed by domain expertise and heuristics.
!!!!Comments & Questions
*One evaluative criteria absent from this paper is the amount of preparation/overhead, particularly with respect to laboratory studies. Simulating historical discoveries in a controlled or exploratory lab setting will still require preparation of materials and tasks, and a domain expert to evaluate the results.
*Can a compliment of methods be used in the setting of VA/~InfoVis? Study of a historical discovery process (research papers, interviews, lab notes), an exploratory/phenomena-data-driven lab study to reproduce the discovery in a simulated setting, a controlled/hypothesis-driven lab study to reproduce the discovery in a simulated setting.
**Can the visual analytics process be carried out by a computational model? For a discovery process that is visually-driven, can you use an intelligent agent with computer vision to simulate the discovery process? The works they cited including computational models were numerological rather than visual data.
*While they mention the development of expert systems and computer-assisted scientific discovery in the motivation, they do not discuss how these methods have contributed to their development in this article.
*The notion of surprise (in the context of unexpected phenomena in a scientific investigation) is closer semantically to serendipity (recognition + sagacity, the ability to act on the unexpected information - <<cite Andre2009>>), than it is to insight, which can relate to both the expected and unexpected. The insight-based methodologies of <<cite Saraiya2004>>, <<cite Saraiya2006>>, <<cite North2006>>, <<cite Yi2008>>, and <<cite Saraiya2010>> all fit nicely into this notion of complementarity in investigation the scientific method, however focused on the moments of insight. Meanwhile, <<cite Mayr2010>> focuses on problem solving strategies.
!![Chang2009] - Defining Insight
<<cite Chang2009>> discerns between two definitions of insight and the implications for these two definitions for the design and evaluation of visualization and VA tools. One arises from the cognitive science literature: that insight is an event, a "eureka" or "a-ha" moment, a change of paradigms and brought on by looking at a problem in a new way. They have neurological evidence for different brain activation during these moments of spontaneous insight. The other definition arises from VA and visualization research that characterizes insight as a quantity, an amount of knowledge gained that occurs upon integrating and building upon one's existing representations, making associations between disparate concepts. While neither is trivial to track or measure, the authors suggest that in the context of visualization and VA, the two forms of insight, however distinct as they are, support each other, occurring in a loop, wherein the knowledge-based insight elicits or enables event-based insight.
!!!!Comments & Questions
*If one form of insight can be measured, the other can be inferred?
*A good starting point into more of the cognitive science literature on insight; insufficient detail here - nothing eye-opening. (citing a New Yorker article?)
!![Endert2012] - Semantic Interaction for ~InfoVis
<<cite Endert2012>>'s CHI paper on semantic interaction. Manipulating the spatial layout via semantic interactions (searching, highlighting, annotating, repositioning nodes), rather than manipulating the layout parameters / algorithms; system parses and interprets user intentions and updates the underlying model as a result. User actions need not be formally defined / require to fit within a formal analysis.
!!!!Comments & Questions
*claim that no current system supports both foraging and synthesis in sense making - true? claim that their system does - entirely true or partially true?
*An interaction technique for DR-na峥rs (mathematical model na峥rs), unifying the foraging and synthesis processes of sense making at the expense of turning the visual encoding and layout into a black box - how transparent are updates to the layout?
*Implications for [[Overview]] - sense making in data journalism
*Implications for teaching and training visual analysts - a trail of interactions and their effect on the data being analyzed, good for professional reflection, finding biases
*Non-semantic interactions - where does panning / zooming / callout selections fit?
*Are there disadvantages to manually manipulating the spatial layout? Memos and implicit knowledge would need to be explicitly recorded for transparency, instructional purposes
*//soft data// and //hard data// confusing terms
*How do new entities (searches that turn up no results, annotations containing new terms) affect the layout initially? Or does layout changes only happen later with dragging, pinning, and linking
*Why do importance values decay over time? How much time passes before this happens?
*Need a full evaluation with a real data set (or a VAST dataset?), multiple users, think-aloud protocol or a means for explicitly recording implicit rationale for semantic interactions
**A possible longitudinal [[Insight-Based Evaluation]]?
*In sensemaking loop, where does semantic interaction occur? throughout? Figure 13 could have been annotated to indicate this
*MS ref: Felger, W. and Schroeder, F. //The Visualization Input Pipeline - Enabling Semantic Interaction in Scientific Visualization//. Proc. Eurographics, 1992
>//Abstract//: Scientific Visualization systems are primarily output-oriented, Users can specify and change parameters that are controlling the visualization process, which will result in different data representations or images respectively. But no mechanism is provided to really interact with the application data (semantic interaction) that has been changed step by step by the process of visualization. In this paper general concepts are elaborated and presented to achieve semantic interaction in dataflow environments for Scientific Visualization.
!![Shipman1999] - Formality Considered Harmful
''Formality considered harmful'': the <<cite Shipman1999>> CSCW journal paper cited by <<cite Endert2012>>, discuses the problems associated with abstractions and formalisms imposed by software: cognitive overhead, tacit knowledge, premature structure and situational structure. While it can apply to VA, tools for information foraging and sensemaking, the paper is grounded in then CSCW space, discussing applications in groupware, knowledge-based systems, computer-assisted design tools, and general-purpose hypermedia. In each example, there are too many formal requirements for naming, chunking, linking, and labelling information and its interconnections.
Formalisms should appear incrementally, and be flexible as tasks shift and change over the course of time (reconceptualization). Users should not have to express knowledge that is tacit - this negatively affects productivity (introspection structures and changes the task, being detrimental to performance). Also designers should consider what is the lesser evil: informally defined information or premature formalization, mischaracterizing the information. Ambiguity and imprecision can be used productively, particularly in collaborative settings. Additional words of wisdom:
*Predevelopment ethnographic study can inform design by capturing tacit knowledge
*Restrict formalization to optional features of the system
*Support incremental formalism
*infer structure through heuristic recognition of textual, spatial, temporal, or other patterns
*Consider the tradeoff between formally-defined optional features and expected frequency of use
!!!!Comments & Questions
*cited by <<cite Endert2012>> in the sensemaking / VA context, which follows the suggestions of formality considered harmful, but the inferred structure via heuristic recognition is limited to the updating of weights
!![Eisenstein2012] - Browsing an Unstructured Text Visualization (WIP)
<<cite Eisenstein2012>>'s CHI WIP poster details an interactive visualization tool for exploring text document datasets via directed search. Of potential use when performing an academic literature search within an unknown topic area, exploring a citation graph. Uses a hierarchical probabilistic topic model to determine the relative topical content of a document, generating topic keywords. The user interface emphasizes directed search and analysis of a small subset of documents and keywords, rather than a low-dimensional embedding of all documents in a collection. The results of a query (author, date, title, keyword, etc.) returns a set of documents, which can be dragged into the canvas, which will arrange themselves into a force-directed layout. Keywords corresponding to these documents will also appear within the layout. The user can toggle between "pinning" the documents or the keywords, meaning than if a set of documents are pinned, they can be moved freely while the keywords respond in the force-directed layout in a "dust-and-magnet" fashion. If keywords are pinned, the converse action is supported.
!!!!Comments & Questions
*No domain aside from academic literature search specified: what about journalism, digital humanities?
*exploration ⥣ted search: how does this application support an initial overview; before the user knows //what// to search for?
*Not immediately clear if topics are computed on-the-fly for these document sets, pre-defined topics:
>//"The topic names are specified in advance, either by a domain expert or automatically; the user is free to rename topics with more familiar terms."//
*distinguishing [author/publication/venue] topics over time / is an interesting and broad use case; may be particularly useful for those in the digital humanities;
*re: other text document visualization tools:
>"//presentations are usually static and display the entire collection at once. This can successfully convey the high-level structure of a collection, but it is poorly suited for more specific tasks//"
*arguments: 1. only a few dozen relevant documents per task/query (true?); 2. low-dim embedding = loss of information; question of what gets ignored dependent on sensemaking task (but what are these tasks? when is it OK for information to be lost? when is it not?)
!![Jianu2012] - UI changes effects on social data analysis
<<cite Jianu2012>> report the results of a study examining the effects of changing nonfunctional UI features in a network visualization application on scientists' hypotheses and biases. These changes involved adding a social update widget (what other users were doing / how many hypotheses were being considered by others), indicating recency of considered hypotheses (via colour coding), and changing the wording and salience of the list of competing hypotheses. Their goal was to decrease biases in hypothesis generation and selection: satisfying, confirmation bias, and failure to consider alternative hypotheses, also the prominence effect. These "nudges", introduced in a counterbalanced within-subjects multi-session experiment, were based on theoretical advances in Persuasive computing and ''libertarian paternalism''.
!!!!Comments & Questions
*"scientifically inspired analysis task"
*libertarian paternalism:
>//"it attempts to influence the choices of affected parties in a way that will make choosers better off//"..."//There has been much critique of the term "libertarian paternalism". For example, it fails to appreciate the traditional libertarian concern with coercion in particular, rather than with freedom of choice in a wider sense.[6] It also aims to promote wellbeing, when there may be more libertarian aims that could be promoted, such as maximizing future liberty//" - wikipedia
*Social vis exploration / interaction: conformity effects and motivational factors induced by observing others' interactions: could be an attractive feature for Linksplorer / Overview (when integrated into document cloud)
*Quant. study - interaction log analysis + qualitative study of workflows (strength of the paper)
*Experimental task inspired by real science, but ultimately artificial; novice ugrad users only - not reimbursed - nothing at stake; toy dataset
**novice effect on task understanding (would not occur with experts); total analysis time was capped, so variation in analysis time isn't very informative
*"//subjects gathered slightly more disconfirming evidence than confirming evidence//" - novice behaviour?
*Network vis view used Google Maps UI for navigation - is this commonplace enough to adopt widespread in visualizations? Is it an expectation?
*UI changes not to vis itself, but to information / metadata panel;
*They could have introduced nudges into different "in-the-wild" deployments in real tools, studied experts longitudinally;
*experts and social nudge - will there really be enough experts working on the same / similar dataset in the same tool at the same time to necessitate this feature?
!![Budiu2009] - Remembering Things Tagged
<<cite Budiu2009>> 's CHI paper compares click-based word-selection tagging against type tagging. A model of human performance and empirical study demonstrated that more tags were created when tagging was performed via clicking on words within the text being tagged, as type-tagging requires a context / modality switch. Less time was spent reading and tagging when tagging was performed via clicking in-line with the article. A memory model and eye-tracking data predicted that recognition would be higher for click-tagging, since there were more fixations on words in articles to be tagged. However, it was predicted that recall performance would be higher for type-tagging, as this form of tagging requires elaboration and personal abstraction. The study compared recognition and recall performance between the two types of tagging as well as with no tagging for content presented in a series of articles. Recognition accuracy was lower for those in the type-tagging condition, while there was no difference between those in the no-tagging and click-tagging groups. Recognition time was lowest in the click-tagging condition. There was no difference in the amount of facts recalled between the 3 conditions. Recall efficiency was poorest in the type-tagging condition. The authors discuss that click-tagging is a "bottom-up" process while type-tagging is a "top-down" process. Their browsing interface supports both forms of interaction. Type tagging encourages elaboration and abstraction, however the interaction / modality switch cost is high, negatively impacting recall and recognition performance.
!!!!Comments & Questions
*recall/recognition study tasks unrealistic: (recall: e.g. "one of the passages you read was about Christmas and Santa Claus": remember and type as many facts as possible in 1.5 minutes); recognition: true/false statements
**an alternative recognition/recall task could display the text again, prompting for a re-tagging, or select from a multiple choice list (recognition), containing their original tag
*huge loss of data from a power failure in their user study resulted in a dramatic loss in experimental power for one of their dependent measures (recall). Why not repeat the study?
*CHI paper bingo: model of human performance re: time/interaction tradeoff, eye-tracking data, application design, A/B/C user study with recognition/recall component, new model to explain results, new memory model, discussion.
!![Grimmer2011] - ~Meta-Clustering
<<cite Grimmer2011>> discusses computer-assisted clustering (CAC), as opposed to fully-automated clustering, a method for exploring a space of alternative clusterings. They primarily discuss unstructured text data, however the process can be applied to numerical data as well. A case study in social science / political science domain illustrates the effectiveness of computer-assisted clustering and the use of visualization for clustering alternative clustering methods (meta-clustering!). The meta-clustering is based on several metrics for comparing clustering results between methods.
CAC routinely outperforms domain expert classifier in identifying new clusters / emergent concepts. It also identifies regions of clustering space that lie within the convex hull of individual clustering techniques, and yet where a single technique does not reveal the interesting or useful clusters identified by the weighted average of multiple clustering techniques.
They perform an evaluation of cluster quality by comparing document pairs with inter-coder reliability and discovery quality by means of several small case studies.
!!!!Comments & Questions
*//some in the statistical literature have even gone so far as to chide those who attempt to use unsupervised learning methods to make systematic discoveries as unscientific// - Armstrong JS (1967)
*+ uses ~MTurk for assessing cluster quality, compares to locally-hired graduate student participants
*in the evaluation, the authors decide on the best clustering methods (or combination of methods) from the meta-clustering, which they argue biases the study against their method, as it doesn't allow the participant to select their own best clustering
*much of the technical detail left to the supplementary material
*yet heavy on technical vocabulary
*links to datasets helpful, but where is the link to the R package?
!![Marchionini2006] - Searching vs. Browsing
<<cite Marchionini2006>>'s communications of the ACM article gives a high-level summary of the differences between lookup and exploratory search, the latter subsuming learning and investigating tasks. The gist of it is in Figure 1:
*''Lookup'': fact retrieval, known item search, navigation, transaction, verification, question answering
*''Exploratory Search / Learn'': knowledge acquisition, comprehension / interpretation, comparison, aggregation / integration, socialize
*''Exploratory Search / Investigate'': accretion, analysis, exclusion / negation, synthesis, evaluation, discovery, planning / forecasting, transformation
Some case studies /examples follow (Open Video Digital Library video browsing and Relation Browser faceted database search). Discusses the new costs associated with exploring (as opposed to lookup): exploratory search offers more opportunities for diversions and advertising on the web.
!!!!Comments & Questions
*a high-level 6-page summary, not delving into HCI / IR fields
*highly web-centric, focus on the net generation
!![Heer2007a] - Collaborative Sensemaking
<<cite Heer2007a>> is a Communications of the ACM version of an earlier 2007 CHI paper, discussing [[sense.us|http://vis.stanford.edu/papers/senseus]], a web-based asynchronous collaborative visualization application for exploring US census data. The application allows view sharing, doubly-linked discussion comments, view bookmarks and navigation (views have unique URLs), and graphical annotation and pointing, altogether allowing an interleaved visualization-based exploration and social activity-based exploration. The application was conducted in several phases, first piloted with an internal research group, then internally within the organization, then over a corporate intranet for several weeks. Usage and comments, as well as bookmarks and links, were tracked over this period. The usage data revealed patterns of comment types (questions, hypotheses, clarification, humour/comment), exploration paths (visualization as primary path), annotation usage (mostly arrows).
!!!!Comments & Questions
*TM: I thought this was an instance of where users in a deployment study used the application differently from what it was designed to do. This doesn't come across in the paper.
*Observed business-like social interactions and comments, an artefact of deployment on a corporate intranet: ManyEyes or broader web deployment would have been more representative of actual social interaction and behaviour.
*future work to integrate Scented widgets into UI
!References
<<bibliography>>
!![Lam2011] - Seven Scenarios of Eval.
<<cite Lam2011 bibliography:Bibliography>> conducted an extensive literature review of over 800 papers published at [[Information Visualization]] venues (~EuroVis, ~InfoVis, IVS, & VAST), over 300 of which reported one or several evaluation methods. Their aim was to study linkages between evaluation goals with methods. The methods reported in these papers, along with their associated goals and research questions, were categorized with 17 tags using open coding. These tags were subsequently reduced to seven distinct evaluation scenarios.
The purpose of these scenarios is to descriptively, rather than prescriptively, guide readers in their own research process, as each scenario is defined by common research goals and questions, along with several evaluation methodologies. Several examples from the literature are used to support each methodology, illustrated by their own goals and research questions.
The scenarios are relevant to all stages of visualization development: pre-design, design, prototype, deployment, and re-design. As shown in the following table, methods may appear among several scenarios, and therefore at different stages in the research. The motivation for this work stemmed from previous reviews on this topic, many of which proposed taxonomies of methods, rather than a categorizations of scenarios. Also, how to conduct the methods is well-reported, but when and how to choose between them is not. The authors argue that a division by scenario is a more natural fit with the research process, as methods in themselves do not map to specific scenarios (research goals and questions). Consideration and selection of research methods should occur only after a goal is defined and a suitable scenario matched to the goal is selected. Evaluation design and analysis is the final step.
|Scenarios proposed by <<cite Lam2011>> (adapted from Table 4, p.18)|c
|!Category |!Scenario |!Example methods |!Distribution |
|process of data analysis^^1^^ |evaluating environments and work practices |field observation, interviews, laboratory observation |process total 15% |
| |evaluating visual data analysis and reasoning |case studies, controlled experiments | |
| |evaluating communication through visualization |controlled experiments, field observation and interviews | |
| |evaluating collaborative data analysis |heuristic evaluation, log analysis, field or laboratory evaluation | |
|assessing vis. use^^2^^ |evaluating user performance |controlled experiments, field logs |27% |
| |evaluating user experience |informal evaluation, usability test, laboratory questionnaire |21% |
| |automated evaluation of visualizations |algorithmic performance measurement, quality metrics |37% |
^^1^^ Holistic view of user experience. ^^2^^ Focus on design decision, explore design space, benchmark against existing systems, discover usability issues.
The authors note that the first category relating to process is grossly underrepresented in the literature despite the importance of the associated research goals and questions, relative to the impact of the questions posed by the second category. The authors emphasize the higher practical value of the first category of scenarios, which extend beyond individual visualizations, techniques, and algorithms.
!!!!Comments & Questions for Discussion
From the discussion section of the report:
*Why are process-scenario evaluations not reported as frequently in the literature?
*Are process-scenario evaluations under-represented due to their typical involvement of methods that take longer to complete (field and case studies) and require extensive qualitative data analysis?
*Is the ~InfoVis community less receptive of process-scenario evaluations, of qualitative research?
**Which communities are more receptive to process-scenario evaluations / and qualitative research. Should the ~InfoVis community aim to be more like them or concentrate on the second category as it has in the past?
!!!!!Further discussion:
*''Categorization'': do design-stages taxonomies (see Table 1, p. 4), miss the mark? As each of the seven scenarios are typically relevant to more than one design stage, does it matter if the taxonomy begins at the design stage (as in Andrews2008), with each stage being associated with one or more scenarios? Is a design-stages taxonomy less appropriate in ~InfoVis contexts (relative to traditional [[Human-Computer Interaction]] studies)? Are stage-based taxonomies more appealing due to their linear flow from beginning to end of a research project? Conversely, are scenario-based taxonomies more realistic as the research process is cyclical and iterative, with scenarios appearing at several stages? Can the taxonomies be used in parallel to converge on a method?
*''Scenario Importance'': the scenarios are not ranked by importance in any way, but there is an implicit importance given to process-related scenarios. Between and within both categories of scenarios - is the relative importance of a research goal determined on a case-by-case basis? i.e. among visualization assessments, which of these take precedence: user experience vs. user performance vs. algorithm performance? In other words, what are the relative values of contributions according to this scenario taxonomy - as they are currently vs. what they should be?
*''Exploratory analysis'': re: scenarios 2,3,4: - its process and outputs are difficult to quantify. What combination of methods allows you to measure when specific insights have been reached by individuals and groups. What methods allow you to determine when individuals and groups attain a higher level of comprehension?
*''Prescriptive vs. descriptive'' taxonomies? Is it possible to avoid selecting methods on a case-by-case basis using a parallel scenario and stage-based taxonomy?
*re: section 1: When to choose [specific research] methods vs. How to choose methods? Are these separable questions?
*''Conducting meta-reviews'': re: section 5.2 - 'open coding and tagging': grounded theory approach and the identification, iteration and final selection of tags. Could tags have been selected a priori? Pros / cons of this approach?
!!!!!Infovis group discussion:
*First six scenarios are user-centric evaluation, the final scenario is visualization-centric. Is there no data-centric scenario?
**Alternatively, is characterization of different data sets not evaluation? Using different datasets for setting benchmarks fits into Scenario 7 for algorithm performance and quality metrics, but characterization of the data itself is not evaluation.
**Table 3's tags for Scenario 7 too broad (only referring to algorithm performance and quality, but not other metrics (clutter, visual ordering, data/ink ratios, etc).
**See Wickham (sp?) paper - verification of clusters and subsets compared to null datasets
**Did paper-selection eval criteria filter of many potential scenario 7 papers?
*The paper itself could use information graphics / visualization to relate quantitative information regarding the number of papers, tags
*The paper's sources only count up until 2008 - have trends in evaluation changed in the past 3 years?
*Why did the authors not include ~SciVis venues (likely to skew distribution of papers using Scenario 7 eval)? ~InfoVis papers at CHI (with more focus on eval)? Note: ~EuroVis is a mixed venue of ~SciVis, ~InfoVis.
*Discussion of differences between heuristic eval, cognitive walkthroughs, expert eval, other discount usability methods
**Methodology terms to detailed for ~InfoVis audience?
**See 2008 paper on [[Heuristic Evaluation]] in ~InfoVis
*Supplementary material needed: the original dictionary of terms, original 8 tags, the full list of papers and the subset of papers meeting their criteria
*Disconnect of Table 1 from text in section 4, section 2 from remaining text
*Descriptive vs. Prescriptive taxonomies; read / revisit other taxonomies in Table 1 - no need for editorializing methodologies in this paper
!![Carpendale2008] - Eval. in ~InfoVis ~Meta-Review
Appearing as a book chapter in //Information Visualization// (A. Kerren et. al. (Eds.) (2008)), <<cite Carpendale2008 >> makes the claim that current evaluations are not convincing enough to encourage widespread adoption of information visualization tools". Drawing from literature in [[Human-Computer Interaction]], perceptual psychology, and cognitive reasoning research, the author describes a variety of quantitative and qualitative methods to be used independently or in conjunction when evaluating information visualization tools. It is hoped that by broadening the repertoire of evaluation techniques, and through the use of multiple evaluation techniques, that claims made about techniques and tools will be more fully reinforced, thus improving their adoption.
The author outlines the challenges inherent in evaluating information visualization tools, namely ''non-linearity'' (decomposing the system into component parts cannot be done), ''holoarchical'' (systems are mutually inter-nested), and ''internal causality'' (system is self-organizing, characterized by goals, feedback, emergent properties, and surprise). In addition, information tools require additional consideration related to their intended purpose of shedding light and insight into large amounts of data, whether or not the tool can "answer questions you didn't know you had", in other words, evaluation methodology should include insight-based metrics to address the exploratory nature of discovery afforded by these tools. Evaluation must also address the perceptual aspects of visualization.
The author proceeds to survey 8 classes of method with regards to ''generalizability'', ''precision'', and ''realism'': field study, field experiment, laboratory experiment, experimental simulation, judgment study, sample survey, formal theory, and computer simulation. There is a subsequent division between [[Quantitative Methodologies]] and [[Qualitative Methodologies|Information Visualization Evaluation: Qualitative Methods]], surveyed and analyzed separately.
In summary, more evaluation is needed in information visualization research, and a broader thoughtful application of a variety of empirical methodologies. The methodology should be a good fit to the research question and goals (but how to map between them is not addressed in this paper).
!!!!Comments & Questions
*The author's suggestions for planning a research process are less direct than <<cite Lam2011>>. It begins with starting with a question or questions that will benefit from further study, relating the questions to existing ideas, theories, and findings. Then one must proceed to identifying an appropriate method for the content, idea, and situation. The leap from research question and situation to method is a black box in this article. When and how to choose among methods is left to the reader. The author does manage to qualify each quantitative and qualitative method with regards to ''generalizability'', ''precision'', and ''realism'' (a discussion absent from <<cite Lam2011>>).
**The taxonomy is a division between quantitative and qualitative methods, and is not sensitive to the research question, design stage, scenario, data, or evaluation goal
*The survey of methods and how they are applied is fairly high-level. The target audience is likely one who is unfamiliar with [[Human-Computer Interaction]] methods, as introductory HCI courses cover all of the methods mentioned. I am surprised at how the methods are not described with examples or scenarios specific to the use of information visualization tools.
*The paragraph on information visualization heuristics is not described in sufficient detail without referring to the original sources.
*re ''Qualitative Methods'' in section 5.2.3, see [[Action Research]] (Lewin C. (Ed.) (2004)).
**''Transferability'' rather than ''generalizability'' when thinking about the concepts of ''reliability'' and ''validity''.
*There is growing recognition that information visualization evaluation is difficult.
!![Andrews2008] - BELIV position paper
Another taxonomy of evaluation in information visualization is offered by <<cite Andrews2008>>, which was cited in the related work table of <<cite Lam2011>>. This short paper from the BELIV workshop provides a list of methods, both qualitative and quantitative, categorized by design stage and purpose along on axis, and class of method based on who performs the evaluation along the other axis:
|Taxonomy of evaluation methods by <<cite Andrews2008>>|c
|!Design Stage / Purpose |!Inspection methods (specialist evaluators) |!Testing methods (representative users) |
|Before design / Exploratory |N/A |[[Observational Study ]] |
|Before implementation / Predictive |[[Action Research]] |N/A |
|During implementation / Formative |[[Heuristic Evaluation]], [[Guideline Checking]], [[Cognitive Walkthrough]] |[[Think Aloud Protocol]] |
|After implementation / Summative |[[Guideline Scoring]] |[[Formal Experiment]], [[Questionnaires]] |
!!!!Comments & Questions
*Observational studies can also be summative if performed after implementation
*Author's opinion: [[Think Aloud Protocol]] and other formative evaluation should receive mention but no more than that (i.e. as part a checklist); "they neither offer validation of an approach nor provide evidence of the superiority of an approach in a particular context"
*Author's opinion: formal studies offer the only objective solutions\
!![Ellis2006] - Formative/Explorative vs. Summative Eval.
<<cite Ellis2006>> observed that among 65 describing new visualization techniques (paper selection criteria is unclear), a small subset mention evaluation (at least plans to evaluate in future work), and a smaller subset of this manage to conduct an evaluation, and an even smaller subset conducts a good evaluation. Many papers justified the effectiveness of their technique by providing visual examples.
The take-home message from this work is that evaluation should be regarded as explorative, rather than summative or formative, which are often not revealing or valuable when viewed from these perspectives. Often these evaluations are necessary for good product design, but they do not constitute good research, which is exploratory. Often authors of research papers include an evaluation even if the result is a foregone conclusion (i.e., even without a study, the best method is obvious) or if even when the evaluation result was not the goal of the research initiative. Other times, the evaluation would address something else, a technique or human behaviour, but not the visualization system itself.
They admit that evaluation of visualizations is difficult due to their complexity (many features), the interaction (independent of the visualization techniques), the diversity of users, tasks, and datasets. Regarding users, they suggest that finding who a visualization technique might be useful for is more important than making the technique work for everyone. They also tackle the common pitfalls of data reporting and statistical analysis and interpretation, and also of experimental design and the selection of appropriate control conditions.
Their big issues revolve around what they call the generative nature of visualization and the lack of clarity regarding the purpose of visualization. They argue that evaluation is complementary to justification leading up to the work (i.e published work, empirical data, expert opinion). Weaknesses in one's evaluation can be addressed by appropriate justifications and vice-versa. Empirical evaluation of a generative process is unsound, but empirical evaluation with reasoned justifications is stronger, more valid and reliable.
The authors comment on what constitutes a good evaluation, one in which results can be generalized, and when appropriate parameters are controlled and subsequently tested. They argue to carefully consider the purpose the evaluation will serve (i.e. whether, when, where, and with who will a technique work, rather than: will the technique work? or is this technique the best?). One must also consider measures, tasks and processes and their mechanisms, how to deal with unpredicted results (i.e. failing to reject the null hypothesis, mixed methods (qualitative and quantitative).
!!!!Comments & Questions
*A very readable, entertaining paper, very much an editorial piece
*Not much guidance as to when you should evaluate, or to exactly how, given your circumstance. It is purposefully a high-level opinion paper
*Discussing the wealth of qualitative studies seemed like an afterthought
*There's no relating this to a concrete taxonomy or design stage. Exploratory studies are never explicitly described as pre-design. It read as if exploratory research at all design stages constituted good research, whereas summative or formative evaluation did not constitute (novel?) research.
!![Sedlmair2011] - Eval. in Large Companies
<<cite Sedlmair2011>> conducted a meta-analysis of earlier research projects (design and evaluation) conducted in a large industrial company setting. The paper serves to present "lessons learned" from designing visualization systems for and with representative users in a large company, a list of common challenges, case studies, and a list of recommendations for overcoming these challenges. While these may not generalize to all company settings, it is likely that many researchers working collaboratively with mid-size and large companies will face many of these challenges.
|Challenges affecting information visualization researchers' work in a large company setting <<cite Sedlmair2011>>|c
|!description | !pre | !during | !post |
|>|>|>| //study/application design// |
|integrating tools in daily work processes: technical, political, and organizational issues | | x | x |
|getting access to the data: distributed, aggregated data, data from everyday work practices | | x | x |
|>|>|>| //participants// |
|choosing domain expert participants: balance b/w unobtrusiveness and intervention, spontaneity | x | x | x |
|getting time from domain experts: especially long-term studies | x | x | x |
|attachment to conventional techniques: users able to factor in time costs with existing tools. Comparative studies difficult | | | x |
|>|>|>| //data collection// |
|confidentiality of information / data (qualitative and quantitative, IPR restrictions) | x | x | x |
|understanding complex work processes: pre-design studies difficult where work is distributed | x | | |
|>|>|>| //results// |
|convincing stakeholders: different from convincing the research community: time and money, not interested in how | x | x | x |
|difficulty publishing your research | x | x | x |
The paper also documents which evaluation methods were conducted during pre-design, during-design, and post-design stages of the earlier projects, how they complemented each other, and the degree to which each was effective:
*''pre-design'': interviews, user observation and [[Contextual Inquiry]], [[Questionnaires]], focus groups (clarifying inconsistencies)
*''during design'': design workshops (validating basic concepts), paper prototyping and mockup reviews, test user review (continued with long-term studies), dedicated usability tests, [[Heuristic Evaluation]], [[Think Aloud Protocol]]
*''post-design'': pair-analysis sessions (for preparing to integrate the tool into daily workflows), quantitative study with domain experts (not for research insight but to convince stakeholders: increased efficiency, decreased errors), long-term studies (integration into daily workflows and data sources).
*informal collaborations, spontaneous conversation throughout
Finally, the paper presented a series of recommendations for collaborating with large industry partners and conducting Information Visualization research in these contexts:
|Recommendations for conducting information visualization research in a large company setting <<cite Sedlmair2011>>|c
|!description | !pre | !during | !post |
|>|>|>| //study/application design// |
|resolve technical barriers to tool integration with domain-specific techniques and tools | | x | x |
|find a hassle-free way to use the right data / avoid additional steps | | x | x |
|choose study environment carefully: find a specific target group rather than general solution at first | x | x | x |
|acknowledge employee-pull and researcher-push solutions to design | x | x | |
|avoid eye-candy, increase usability, functional aesthetics | | x | |
|provide tech support, allow easy installation | | x | x |
|>|>|>| //participants// |
|restrict study sessions to one hour | x | x | x |
|convince the right target audience: use the right target data/problems | | x | x |
|learn from the experts, study their processes in detail: avoid novelty, identify what makes their processes effective | x | | |
|usability tests should be done with outside users (save experts' time) | | x | |
|send gentle reminders to study participants (esp. in long-term studies), informal meetings | | | x |
|>|>|>| //data collection// |
|get IPR licenses, do studies in any case, provide privacy in qualitative studies | x | x | x |
|be in constant close cooperation, be flexible and spontaneous | x | | |
|>|>|>| //results// |
|show how your technique will save them money / increase profits, rather than display research novelty | | | x |
|factor in high skill w/ current methods: factor in learning, reluctance to adoption | | | x |
|clarify publication intentions up front | x | x | x |
!!!!Comments & Questions
*An impressive amount of research over a 3.5 year period. Excellent recommendations
*I'd be curious to understand barriers and recommendations for mid-size or smaller companies, esp. research start-ups or mid-size companies with active research divisions. Or healthcare / pharmaceutical companies.
!![Plaisant2004] - Challenges of ~InfoVis Eval.
<<cite Plaisant2004>> discussed common challenges and barriers to evaluating information visualizations:
*a need for realistic users, tasks, domain problems
*improved user testing: adequate training (or no training if tool learning is to be evaluated), feature usage, a greater breadth of tasks (beyond time and error), reporting results per task rather than global results, addressing the generative, exploratory nature of visualization use, insights and hypotheses generation, acknowledging that visualization use is situated in a work context along with other tools and processes.
*address needs of atypical users: universal usability
It advocates the creation of benchmark task and data sets for future evaluation (type of evaluation not specified). It also calls for more case studies to be reported. Finally, toolkits for creating visualizations should be expanded with a breadth of current techniques and features, many of which have been evaluated with documented results. The paper is concluded with technology transfer examples.
!!!!Comments & Questions
*While this paper not a meta-analysis of the evaluation literature, nor was it a description of evaluation techniques, its contribution was to define the challenges. Recommendations for design and development of evaluations are at at a high-level only.
*re: the technology transfer examples - does this signify that if one's tool is similar in functionality or tasks to a similar tool that has been adopted, this constitutes as evaluation? ("we are still a long way from understanding how our research tools become products").
**this section seems tangential - connections to evaluation methodology are unclear.
*Overall, underwhelming after reading more recent BELIV papers and other recent publications. Addressing design stages, evaluation goals, research scenarios are absent; recommendations are without specific evaluation techniques
!![Chen2000a] - Early Eval. in Vis.
<<cite Chen2000a>> is an early meta-review of evaluation in information visualization circa the late 90s, surveying 9 previous studies. They stress the need for evaluating new techniques in practical contexts, mentioning that user-centred design is rare in practice.
!!!!Comments & Questions
*An insufficient amount of detail is provided about these previous studies, so at this point in my review the paper is not very helpful.
*They only mention the bottom-line finding of these studies (i.e. technique x outperformed technique y, users performed technique z, etc.)
!![Thomas2005] ch.6: Moving Research into Practice
[>img(33%, )[three levels of evaluation metrics|http://hcil.cs.umd.edu/trs/2004-30/2004-30_files/image004.jpg]]
In the second chapter of PNNL's //Illuminating the Path// research agenda, <<cite Thomas2005>> provide a meta-analysis regarding evaluation for ~InfoVis and VA tool, up to date and well-aligned with the above research. The chapter's higher-level goal is to address the challenges of moving tools into practice, so in addition to evaluation, they also discuss the need and acknowledgement of privacy and security issues (particularly in homeland security / intelligence settings), the need for interoperability with other tools and workplaces practices, and barriers to technology adoption. Each section of the chapter discusses benefits and costs associated with considering these facets of tech-transfer, and suggest recommendations for future research and deployment initiatives.
With respect to evaluation, they advocate realism (tasks, users, data, objective measures). They highlight the benefits of evaluation (verifying hypotheses, creating common benchmark tasks and data sets, increasing communication between academia and industry, compare alternative approaches, and confirming program goals). They illustrate how evaluation programs have benefited other fields (i.e. natural language understanding / speech recognition / data mining / information retrieval / information visualization contests). They discuss approaches and associated challenges for evaluations at several levels (component, system, and work environment - see figure), in line with other research published in 2005 or earlier.
!!!!Comments & Questions
*Once you read a single subsection of this book, you realize there's a clear formula to each section: here's what's known with regards to subject x / here's what's been done (or what's the state of the art), and what are the recommendations for improving this process or better-integrating the process into further R&D.
*I appreciated the section illustrating how other fields have increased the breadth and value of evaluation techniques via benchmark tasks, data sets, ground truths, often in the forms of contests.
*The problems associated with evaluation techniques at component, system, and workplace levels are captured in more detail by other work up until this date, so I will not reproduce them here.
!![Elmqvist2010] - BELIV position paper; External/Internal Validity
<<cite Elmqvist2010>>'s BELIV '10 position paper suggests conducting 2 summative studies when evaluating an ~InfoVis system. One quantitative study, rigorous but unrealistic ([[Internal Validity]] but not [[Ecological Validity]]): evaluating visual search or a toy data set task-benchmark study, and one qualitative study, realistic but ad hoc ([[Ecological Validity]] but not [[Internal Validity]]). He argues that 2 studies are not necessarily double the amount of work, although conducting two mutually linked studies is non-trivial. They can use the same participants, experimental design, tasks, experimental session and analysis. Thus the amount of time to conduct the studies and the amount of paper space to report the studies is not exorbitant.
!!!!Comments & Questions
*How can a qualitative and quantitative study use (essentially) the same analysis?
*If they make use of a single experimental session, how do you account for training / learning?
!![Kang2010] - Mixed Methods and Triangulation
<<cite Kang2010>>'s BELIV '10 position paper discusses [[Mixed Methods]] approaches, pits usability vs. utility, quantitative vs. qualitative for the purposes of evaluating ~InfoVis tools built for investigative analysis. They discuss pragmatic challenges (to a greater extent than <<cite Plaisant2004>>), including the use of students as evaluation subjects, the balance between qualitative and quantitative, and the issues of statistical power vs. analysis depth, scalability (re: data sets used in evaluation), learnability, task comprehensiveness (exp. for open-ended tasks).
They comment that "Quantitative methods are most appropriate when the outcome is measurable and the purpose is to validate findings. Qualitative methods are better for identifying pain points, best practices, and providing design feedback and guidance."
!!!!Comments & Questions
*Implicitly, they indicate that qualitative methods are best at the formative and summative stages of design, while quantitative methods are best during design. This is not explicit.
*It was nice to see the distinction between evaluation goals stated as the choice to measure system usability vs. system utility. I'd like to see a meta-review paper look at this distinction in past work.
!![vanWijk2006] - Summative Eval. of Vis as Tool vs. Vis. as Communication Medium
<<cite vanWijk2006>>'s essay considers the role of [[Information Visualization]] as a scientific discipline, as technology (improvements in efficiency and effectiveness), and as an artistic/design discipline. He addresses the incremental nature of research in the area over the preceding years, indicating the maturity of the field.
In terms of evaluating the value of a visualization, from a formative or summative standpoint, he proposes an economic model that accounts for the knowledge gained and the costs (time, money, resources) associated with developing and using visualization tools. However, he also addresses the ill-defined nature of knowledge gained and insight in these evaluations, and how it is difficult to account for a priori knowledge of the observer. Nevertheless, several implications are to be considered when developing (or contemplating whether to develop a system, including accounting for alternative methods, initial costs, the subjective nature of visualization, negative knowledge, the costs of interaction (encouraging use of presets, avoiding unnecessary interactions. He emphasizes the often not-considered role that visualization plays in presentation, rather than for exploration and problem-solving, and that its use for presenting is at least as important:
>//"The purpose of visualization is funding, not insight."//
He laments that the number and use of a variety of evaluation/validation methods in the field is still low, and doesn't account for the many ways visualization can be interpreted: as a science (controlled studies), as a technology (automated validation, requirements gathering), as an art/design (design critique). A wider breadth of techniques for evaluation is needed to account for these views. Requirements gathering is often not seen as a creative process in itself, and practitioners should account for this at this stage.Requirements will often be conflicting and will vary in importance, so a weighting is encouraged. Like the calls of <<cite Ellis2006>> or <<cite Greenberg2008>>, rigid evaluation at early stages can stifle innovation and should be postponed. Design should be top-down and cyclic. Continuous development and prototyping is encouraged.
>//"Developing great ideas is simple, rejection of bad ideas takes all the time."// - on the art of visualization
Misconceptions regarding design are also addressed: it is not a matter of talent - design can be learned and practices like any other activity. Design is not about creativity, but often about following conventions. Design is not a matter of inspiration, it is a matter of hard work. Design is not about aesthetics, but about function. Design is not about taste, it is about rational choices. Design should communicate purpose clearly, as with writing papers.
Finally, the field must progress as a science, with increased rigour in empirical evaluations of techniques and human information processing and perceptual capabilities. Taxonomies and hierarchies of tasks, users, data types, types of use, and techniques should account for theories, frameworks, and models grounded in empirical knowledge.
!!!!Comments & Questions
*Recommended article from MS, an extension of a previous conference paper on the value of visualization
*A ~VisWeek 2011 panel addressed whether Visualization should be regarded as a science: "Theories of VisualizationԨere Any?" - the author sat on this panel - he argues that visualization is not a science, but a technology:
>//"we try to develop methods and techniques that enable people to do their job more effectively, efficiently and with greater satisfaction."//
**What customers should use, when and why, are the questions that should be driving the research community.
*Mirrored a recent ~InfoVis group meeting - a need for more design critique
!![Bertini2011] - Quality Metrics
<<cite Bertini2011>>'s 2011 ~VisWeek paper describes [[Quality Metrics]], specifically for [[High-Dimensional Data]] (HDD), at the data and image stages of the [[Information Visualization Pipeline]]. They conducted an extensive [[Grounded Analysis|Grounded Evaluation]] of the literature, of studies that reported the use of quality metrics in HDD cases. They summarize the role of metrics in these papers, what is measured, where in the pipeline measurements occur, what interactions are supported, the types of HDD visualization often assessed by metrics, and the use of [[Meta-Visualization]] to allow selection among alternatives. They review three case studies covering many aspects previously discussed in their review.
''Findings'': They found a lack of consistency in how these metrics were applied. [[Meta-Visualization]] was not universally used, as the means of exploring alternative images generated by quality metric algorithms was not consistent. The ''View Transformation'' stage of the pipeline was the least represented in the quality metrics pipeline, with more metric use reported at the ''Data Transformation stage'' (feature selection, projection, aggregation, sampling) and the ''Visual Mapping'' stage. Metric scalability to larger data sets was also seen to be an issue in the literature. Finally, comparing visual techniques on the same data was found to be an unexplored area.
''Future Work'': User evaluation (utility, usability) or interactive metrics is required. The interaction with human perception is also an area of development for HDD quality metrics. A systematization is needed to effectively compare metrics. Finally, the scalability of these metrics must be addressed.
!!!!Comments & Questions
*A grounded analysis of the literature - not theory-driven?
*papers on HD graphs thrown out.
*related work cites other meta-reviews / taxonomies in ~InfoVis literature: a need for systematization
*examples and case studies come from authors' previous work.
!![Hullman2011] - Visual Difficulties
<<cite Hullman2011a>> advocate a balance or tradeoff between efficient, minimalist design in ~InfoVis with visual difficulties that elicit active processing on the part of the user. The authors call these //desirable difficulties//. Such active processing is associated with higher long-term retention of information, in other words, these difficulties are more beneficial for the purposes of learning. Along with a meta-analysis of instances where a degree of visual difficulties has improved user performance and learning, this paper has several implications for the design and evaluation of visualizations.
''Evidence'': the 'Visual Difficulties' theory addresses the major tenants of the [[Cognitive Efficiency Model of Graph Design]]: these being minimizing cognitive operations, maximizing discriminability and minimizing complexity of visual encodings (within perceptual/cognitive limits), reducing the data/ink ratio, organizing information in a logical way (reducing time to interpret quantitative relationships), using animation to reduce the mental manipulation of complex entities over time, and using direct labeling. 'Visual Difficulties' hinges on the perspective that graph reading is a learning process, requiring 'active processing' and a high level of 'engagement'. 'Visual Difficulties' accomplishes this in the following ways:
*provoking self explanations: stimulating intense cognitive activity, relaying less on automation for problem solving (i.e. macros, saved settings)
*manipulating internal visualizations: using time slice small-multiples rather than animation, relying on internal representations and mental models rather than offloading cognition and allowing the system to display a transformation
*perceptual disfluency: making text or graphics difficult to read forces the reader to rely on peripheral cues:
>//"easy-to-process graphs may actually lead to superficial interpretations"//
*engagement and motivation: novelty (adding chartjunk?), aesthetic appeal, and personalization make graphs more memorable, despite being less efficient than minimalist approaches; gamification is another approach to increase motivation/engagement, manipulation of internal visualizations
''Design Implications'': the authors advocate a tradeoff between visual difficulties, such as those described above, and the [[Cognitive Efficiency Model of Graph Design]], summarized in the following:
|cognitive efficiency vs. visual difficulties <<cite Hullman2011a>>|c
|!Criteria |!Cognitive efficiency |!Visual difficulties |
|cognitive operations |minimize cognitive steps |induce self-directed cognitive activity |
|data/ink ratio |maximize the data to ink ratio |design to provoke active processing |
|organization |important info. most salient |choose format that provokes cognitive reflection on important info |
|animation |quick and intuitive animation |static representations, provoke internal manipulation |
|labeling |labels rather than legends |use legends |
They also emphasize the role of individual differences, prior knowledge, and the possibility that efficient graphs might make novices overconfident. Using multiple representation formats is encouraged. They also admit that visual difficulties will not scale to large datasets. Finally, the authors encourage designs that provoke self-doubt, caution, uncertainty - forcing users to double-check inferences.
''Evaluation Implications'': Like the BELIV proceedings, this paper calls for a wider breadth of evaluation techniques that evaluate more than what [[Cognitive Efficiency Model of Graph Design]] can provide, going beyond response time and error rates. Adopting methodologies to better understand the generation of insight, measuring task transfer and learning (ie. <<cite Chang2010>>'s ~Learning-Based Evaluation), assessing effort (cognitive load), engagement, and level of interest.
!!!!Comments & Questions
*I see how this may apply in textbooks, classrooms, educational software (emphasis on learning here). But hardly appropriate for casual ~InfoVis //OR// expert users: they don't discuss the role of ~InfoVis in decision making (efficiency is key in both cases); Also consider the case of practical info-graphics (i.e. transit maps, task flows, organization directory/ hierarchies: visual difficulties would be discouraged here if the task is quick look-up of information
**gamification seems perfectly appropriate for educational graphics/software, where motivation is to be challenged
**re: efficient graphs might make novices overconfident; the graph reading is only one aspect of the VA process; without domain knowledge and other analytical skills that come with expertise, synthesizing other sources of information, it doesn't matter much if graph reading is as easy for novices than it is for experts
**re: large datasets: we're now out of the realm of education and novices - this is the domain of experts where visual difficulties should not be considered.
**I also feel that some learners will respond well to gamification and novelty while others can absorb more through efficient reading of information without pausing to consider self-explanations: that being said, the authors don't address learning styles! (lumped into individual differences?)
*Motivation and engagement is notoriously low in controlled studies: the participants don't care about the toy task/dataset, so it's no surprise that they don't learn as much. Introducing visual difficulties makes the tasks more challenging and thus more memorable - but when these participants find a task and dataset they //ARE// interested in - they will likely be frustrated by visual difficulties
**the original studies are flawed because the baseline conditions are likely to have zero/very low motivation, whereas in the real world, people will only look at graph if they have the motivation to do so, because the data is of interest to them: they don't need added motivation from novelty, aesthetics, personalization
*is aesthetic appeal (other than the minimalist aesthetic) an agreed-upon notion: it is likely too subjective
*Table 1 vis. difficulties column doesn't vary much in its design suggestions: the first three just choose alternative forms of stating that whatever you design, make sure it makes your users 'think' really hard
*Re: implications for evaluation don't necessarily rely on visual difficulties - a wider breadth of evaluation methodologies and metrics should be adopted regardless.
**As in <<cite Chang2010>>'s ~Learning-Based Evaluation, it is difficult to disentangle what is learned about the task from what is learned about the interface.
*Re: animation vs. small multiples / timeslices (as in PSYC 579, the latter is already superior for allowing parallel processing, less need for interactive control, less reliance on WM)
*~VisWeek 2011 honourable mention
*[[Steven Few commentary|https://docs.google.com/viewer?url=http%3A%2F%2Fwww.perceptualedge.com%2Farticles%2Fvisual_business_intelligence%2Fvisual_difficulties.pdf]]:
**desirable difficulties are not necessarily visual in nature; engagement leads to active processing, not visual difficulties
**self-explanation not a visual difficulty
**visual complexity is required where complexity in the data exists, hardly constituting visual difficulty
**Kahneman, D. (2011). //Thinking Fast and Slow//. Farrar, Straus, and Giroux (eds.): people view difficult-to-read text as less trustworthy than easy-to-read text. Bad for trusting data sources, but good for representing actual uncertainty in the data when it exists - visual difficulties need not be addressed
>//"If they粯ng, however, which indeed they are, their claim could do great harm. An extensive body of research from several disciplines has long directed us to display data as simply as possible, eliminating gratuitous, distracting, and difficult-to-perceive visual content in an effort to reduce unnecessary and unproductive cognitive load."//
>//"long-term recall is rarely the purpose of [~InfoVis]"//
*re: gamification:
>//"People indeed love to be challenged, but not in ways that waste their time by presenting obstacles that aren㥦ul//"
*re: 3D Vis, chartjunk and other novel embellishments:
>//"people too often prefer things that arenﯤ for them"//
!![North2006] - Measuring Insight
<<cite North2006>>'s short editorial article addresses the issue of measuring insight generation in visualization. This is part of his group's ongoing [[methodological research|Insight-Based Evaluation]], designing studies to capture insight: <<cite Saraiya2004>> (controlled insight quantitative experiment), <<cite Saraiya2006>> (qualitative field study), <<cite Saraiya2010>> (comparison of benchmark experiment vs. quantitative insight experiment).
''Insight'', the purpose of visualization. It is complex (how much data is involved), deep, qualitative, unexpected, and relevant. The purpose of evaluating visualization systems should be to gauge how well the system provides insight.
Benchmark user studies, on the other hand, involve predefined tasks having definitive completion times and definitive simple answers. Overly search-like tasks doesn't represent insight. Additionally, results for simple tasks do not scale for complex tasks, as a definitive task hierarchy does not exist. These user studies also involve tradeoffs of speed and accuracy.
To measure insight in a controlled and direct way, or to compare between alternative visualizations, tasks of greater complexity must be used (beyond search/look-up tasks), tasks in which accuracy is not a Boolean result (correct/incorrect). Examples include characterizing distributions, correlations, and patterns, estimating various statistical metrics, etc. The tasks must be carefully worded to avoid bias. These tasks will undoubtedly be longer, there will be greater variability in completion time, and generating isomorphic tasks will be difficult. Additionally, more participants will be required to reach statistical significance.
The other option is to measure insight in a qualitative, unconstrained way, with real domain users, interacting with their own data, to measure what insights users gain on their own. A [[Think Aloud Protocol]] is required, and a rigorous coding protocol for characterizing insights is needed, one that captures insight category, complexity, time to generate, errors, and depth. For this a domain-expert coder must participate. While objective, a rigorous coding scheme and multiple raters will produce more significant results. Once this data is collected, it can be quantified and compared to results generated by other tools, or to what a domain expert would expect to learn from a dataset. These open-ended studies can also act as usability tests, correlating usability issues to insight generation. However, these studies will require long training and trial times, less participants (but domain-specific), and more effort to code the results, and participation of a domain expert in coding. On the other hand, the experimenter no longer needs to design benchmark tasks. It is important that benchmark tasks, if evaluated, be administered after an open-ended insight protocol, such that bias is avoided (narrowly-defined tasks will constrain the thinking of participants).
!!!!Comments & Questions
*A good concise summary of his group's research direction, and the problem with benchmark task user studies
*Insight remains undefined, although some purely qualitative characteristics are generally accepted. Quantifying insight will always be a subjective process, however improved by inter-coder reliability
!![Wagstaff2012] - ML (Or ~InfoVis) that Matters
<<cite Wagstaff2012>>'s ICMI position paper could read the same as "~InfoVis that matters". It discusses problems faced by the field: statistical significance favoured over practical versus, a lack of reproducibility in the field, a lack of depth in the interpretation of results of experiments, an over-reliance on synthetic data and insufficient detail regarding data provenance when data is not synthetic, and application domain weighting of false positives and false negatives. It offers guiding goals for the future of the field, including discussions practical significance in application domains, comment papers from those domains, reducing jargon, mitigating risk factors of application deployments (e.g. dealing with false positives/negatives), and reducing the complexity of abstractions used. Some new challenges of practical significance are offered.
!![Heer2010] - Crowdsourcing Graphical Perception
<<cite Heer2010>>'s CHI paper discusses crowdsourcing as a means of evaluating visualization techniques and tools. Using Mechanical Turk, They replicate previous laboratory experiment results (<<cite Cleveland1984a>>, Stone, M. and Bartram, L. (2009)), generate novel graphical perception results, and provide recommendations for future users of crowdsourcing services:
*On setting up a study/experiment using Mechanical Turk:
**micro-jobs are known as //Human Intelligence Tasks// or //~HITs//. Each trial of an experiment is a single HIT. You can throttle the number of ~HITs based on desired experimental conditions.
**To create ~HITs, you can either use the existing Mechanical Turk markup language, or host your own task on an external site within an embedded frame (the authors host a Flash application in one experiment); in the latter case, you can capture more information about the user, such as their operating system and configuration, using ~JavaScript
**Authors found a significant effect of operating system in the experiment replicating Stone, M. and Bartram, L. (2009)'s scatterplot grid luminance experiment
**Multiple choice qualification tasks that mimic the open-ended experimental tasks are a good way to limit users from "gaming" the tasks, and ensures that users understand the tasks. Expect a 10% data loss due to unusable responses if a qualification task is not used. Describe qualification tasks carefully
**$0.02 (2010 USD dollars) per HIT was seen as an adequate rewards based on expected response time and the 2010 US minimum wage. However, response times varied considerably, were overall longer than expected on average, and thus at the rate of $0.02 / HIT, participants were underpaid
**name your ~HITs such that multiple experimental settings are viewed as options (e.g. bank foo, bank bar), rather than steps (e.g. part 1, part 2)
**Triangulate findings with traditional small-scale laboratory experiments
**Services like [[Turkit|http://groups.csail.mit.edu/uid/turkit/]] aim to support dynamic task generation and easier access control, allowing for adaptive studies, between-subject designs, and prevention of subject overlap across experiments
*On the quantity and quality of responses:
**Raising the reward for each HIT increases the quantity, but not quality of responses.
**Many "Turkers" participated in more than one experiment, or bank of ~HITs, meaning that between-subject experiments may be difficult to conduct without constructing batch tasks / macro-tasks, which is discouraged in the Mechanical Turk community
**There are //streakers// and //samplers//, meaning a set of bimodal completion rates, some Turkers will complete only a few ~HITs and not complete an entire bank, while others will complete multiple experiments' worth of ~HITs; also has implications for within/between-subject experiments
**Timing data is not reliable, nor are timing instructions easy to enforce, short of embedding your own web frame containing an experiment application into an HIT
**expect up to an order-of-magnitude in experimental cost savings (compared to standard $10-$15 lab experiment payouts)
**Process can scale to hundreds of participants within a couple of days, thus saving weeks of experimental administering
*Experimental materials are online here: http://hci.stanford.edu/gp/chi10.zip
!!!!Comments & Questions
*Extremely helpful how-to and lessons-learned paper
*Well-written
*What developments to Turkit have occurred since 2010?
*Do alternatives to Mechanical Turk exist? Can studies be run in Canada?
*Companion paper: <<cite Willett2012>> on strategies of improving social data analysis among crowdsourced participants
!![Kosara2010] - Eval. w/ ~MTurk
<<cite Kosara2010>>'s short 2010 BELIV paper offers additional guidance and some lessons learned from conducting visualization studies using Mechanical Turk. Using this service, experimenters can pay for good performance, recruit many users in parallel, expect faster study completion times, and ensure a higher diversity of participants, when compared to traditional laboratory studies. They too discuss the feasibility of embedding a Java applet into an HIT (Human Intelligence Task). A screen resolution calibration task should be used to ensure that the applet fits into the screen of the user. The Mechanical Turk requester side is fully automated and accessible through a web services API from most programming languages.
They conduct 2 studies with Mechanical Turk, one of which replicating a previous study regarding visualizing graphs (and achieving differing results), and a novel study regarding the visualization of percentages.
They acknowledge the demographic and personality differences between Turkers and student population pools used for laboratory studies. There are more women among Turkers, and they tend to be less extroverted, more neurotic, and more open, imaginative, and curious. However, there is no way of verifying demographic data supplied by Turkers.
Finally, the best time to launch a study is on Monday morning or during the EST weekday lunch time.
!!!!Comments & Questions
*After <<cite Heer2010>>, little more was gained from reading this paper.
*Study 2 (Section 4.2) had no baseline condition (lab study) to compare against, unlike Study 1, which makes me skeptical about the results and how they generalise.
*Small practical guidance useful. Little in terms of novel research contribution emanating from 2 studies (no in-depth reflection).
*Personality difference findings are interesting, but again little reflection is given aside from openness to visual metaphors. How should we take this into consideration when we design prototypes and studies?
!![Meyer2012]
<<cite Meyer2012>>'s BELIV 2012 submission refers to the earlier paper by <<cite Munzner2009>>, the nested model of //domain problem characterization > data and task abstractions > visual encoding and interaction choices > algorithm choices//: 4 blocks of visualization development. They address this to the BELIV workshop in the form of a guidelines for conducting evaluations: within block comparisons and between-block mappings. Within-block comparisons: comparing competing algorithms against one another, or alternative visual encodings or interactions. Between-block mappings: evaluating the suitability of algorithms for a particular visual encoding or interaction choice, or the suitability of a visual encoding or interaction choice for a particular data or task abstraction.
!!!!Comments & Questions
*Editorial comment that technique papers should consider higher levels of the nested model, including domain problem characterization, as should design-study work consider algorithm choices.
*Within-block comparisons are still ill-defined for domain problem characterization and data/task abstraction, as is the mapping between them. The field requires a mid-level task taxonomy to better understand task abstractions.
*Is it possible to compare competing domain problem characterizations?
!![Gotz2009] - HARVEST (adoption code in Lam2011)
<<cite Gotz2009>> (journal version of <<cite Gotz2008>>), coded by <<cite Lam2011>> as an adoption evaluation study; eval of HARVEST VA system not the focus of the article (a action/task taxonomy motivated by Activity Theory); eval of HARVEST used as case study for taxonomy; collected feedback on system from visualization developers and from "several end users":
>"//We also observed several end users of the HARVEST system. The users analyzed enterprise data containing personnel, funding and strategy information describing ongoing projects within our company. In addition to observing their behavior, we performed interviews to gather feedback on HARVESTࡣtion tracking and insight provenance capabilities.//"
Only a single paragraph summarizing initial user feedback is provided (no discussion of adoption).
!!!!Comments & Questions
*How did they become end users? What industry were they working in? How long were they using the tool?
*description of adoption tenuous
!![Kincaid2005] - ~VistaChrom (adoption code in Lam2011)
<<cite Kincaid2005>>, coded by <<cite Lam2011>> as an adoption evaluation study. Eval not a focus of 15p article, only a 5 paragraph "user feedback" section prior to conclusion:
>"//VistaChrom has been in use in Paul Meltzerଡb in the Cancer Genetics Branch of the National Human Genome Research Institute, where it has been an active part of their workflow for over a year. More recently, VistaChrom has been deployed at Mike Bittnerଡb in the Molecular Diagnostics and Target Validation Division of the Translational Genomics Research Institute, as well as a number of early access customer sites, which are evaluating AgilentࡃGH platforms. These labs have included VistaChrom plots in presentations at a recent oncogenomics conference.//"
4 additional paragraphs summarizing user feedback re: design, learning curve, interactivity, familiarity given domain conventions, scalability, simplicity of interface:
>"//The simplicity of the navigation and visualization has been advantageous as a presentation tool for sharing results.//"
Scalability concerns fed back into design and into future work:
>"//From user feedback, we quickly learned about the scalability issues required to analyze tens or hundreds of arrays. This led us to design洨e previously mentioned feature that computationally selects which arrays to display using Z- score criteria, rather than using manual selection based on visual observation.//"
!!!!Comments & Questions
*worth citing in Overview paper; comments on who adopted and for how long, high-level tasks, not much detail on their workflows; comments on how user feedback fed back into design.
*design the result of "working closely with several collaborators"
!![Wang2007] - ~IVoW (adoption code in Lam2011)
<<cite Wang2007>>, coded by <<cite Lam2011>> as an adoption evaluation study;
>//"We have performed user studies to evaluate the effectiveness of our approach and demonstrate that visual analysis can be successfully combined with network security mechanisms to greatly improve intrusion detection capabilities."//
Reports a controlled lab study with 8 volunteers "//who are research staff or graduate students in computer science or electrical engineering majors. Half of the subjects have network and security back- grounds while the other half do not. Most of the subjects do not have graphics and visualization backgrounds. All the eight volunteers have normal vision and are not color blind.//"
Conclusion reads:
>"//To better evaluate the proposed approach, we will apply IVoW to real large-scale wireless network environments such as underwater sensor networks, in which the nodes can move freely in 3D spaces.//"
!!!!Comments & Questions
*no user adoption reported (though the word adopt appears many times in the article, it refers to adopting particular algorithms, techniques)
*reports a user study experiment; why did Lam et al code this as adoption?
!![McKeon2009] - Dashiki (adoption code in Lam2011)
<<cite McKeon2009>>, coded by <<cite Lam2011>> as an adoption evaluation study; describes a prototype wiki application that integrates visualization authoring into wiki pages;
>"//In addition to describing these technologies, we provide a preliminary report on the public launch of a prototype based on this design, including a description of the activities of our users derived from observation and interviews.//"
Authors invited users of Many Eyes to a limited access beta in Nov 2008; opened to public in Feb 2009. Nearly 30K page impressions as of Mar 2009. 3.5% of traffic involved edits in addition to pageviews (6% before public launch); 118 user registrations, 37 have revised or created content; 349 pages created; comments on how users connected data sources to the wiki content created; discusses the three categories of user-generated content based on the 349 pages created; conducted 3 semi-structured interviews:
>"//We conducted three semi-structured voice interviews with three Many Eyes Wikified users, identified through observation of their activity and selected because they were particularly prolific in their use of the system. In conducting these interviews, we sought to identify what it was that drew them to the system, develop a better understanding of the value that it had for them, and provide a context in which they could discuss problems and opportunities for improvement.//
>
>"//By choosing to interview only prolific users, there is of course an inherent selection bias in these narratives that makes it difficult to generalize any conclusions made across a broad population of users. However, we believe that the resulting discussions are still valuable both to inform development of similar systems and to suggest future directions for this research."//
Interviews covered a range of content creation habits; topics dicussed include context and background, page type, feature usage, usability problems / visualization interpretation issues, prior experience with wikis, frustrations, data ingestion issues, appreciation of minimal design, visualization coordination and faceting, input file format, data subsetting and filtering, data export and access control, secure access, integration of wiki content with visualization and datasets.
Conclusion:
>"//We see a clear need to explore how users are reusing Dashiki content, and what additional steps might be taken to improve its utility, compatibility, and communicative power in these contexts//""
!!!!Comments & Questions
*worth citing in Overview paper, examines similar range of design, usability, data ingestion issues in user interviews
!![Robinson2008] (adoption code in Lam2011)
<<cite Robinson2008>>, coded by <<cite Lam2011>> as an adoption evaluation study;
>"//This paper reports the results of collaborative synthesis experiments conducted with expert geographers and disease biologistsᮠexperiment was designed for participants to simulate the real-world task of determining the source of an avian influenza outbreak in the Pacific Northwest.//"
!!!!Comments & Questions
*reports a user study experiment about collaborative activity; why did Lam et al code this as adoption?
*no viz tool in the paper; just a paper artefact sorting experiment, coding activities: annotation, zooming, grouping, tagging, collaborative events
!![Kandel2011]
<<cite Kandel2011a>> on research directions in data wrangling. In //sources of data problems//, data transformation is cited as one class of problems. Research directions discussed include visualizing raw data and providing best practices, scalability and the risks of premature aggregation, outlier detection and selction, sampling, living with "dirty data" (missing, uncertain data), how to indicate missing / uncertain data such that conclusions aren't based solely on remaining data.
Research directions in transforming data are also discussed: "//reformatting, extraction, outlier correction, type conversion, and schema mapping.//"). Many existing tools are not a complete and iterative transformation pipeline, so they are used idosyncratically. Error correction, entity resolution, and data set integration are some of these trasnformations.
Editing and auditing wrangling and trasnformation steps are also discussed: how can these processes be made transparent, understandable, and reproducible?:
>//"we contend that the proper output of data wrangling is not just transformed data, but an editable and auditable description of the data transformations applied"//
High-level languages could be used to produce transformation scripts.
On the need for humans in the loop:
>//"One of the insights motivating our interest in data wrangling tools is that algorithms are not enough. Nuanced human judgements are often necessary throughout the process, requiring the design of interactive tools."//
Sharing data transformation scripts online:
>//"By deploying wrangling tools on the public web, a large audience (analysts, journalists, activists, and others) might share their transformations, and thereby further open data access."//
Domain specific data types could be elicited / crowdsourced from domains. Annotations from downstream analysts could feedback into wrangling decisions (e.g. <<cite Willett2012>>).
Concluding thoughts:
>//"We have argued that data wrangling should be made a first-class citizen in the data analysis process. ɦ we do not address this issue in the near future, we run the risk of disenfranchising the very domain experts on whom we depend for our data analysis."//
Future work: empirical studies that show "//how day-to-day users wrangle with their data//".
>"//Data dissemination (especially by governments) has traditionally been done in the form of printed reports, and there are still data providers that consider a PDF scan of a report a digital version of the data.//"
Paper ends with a summary table of research directions.
!!!!Comments & Questions
*bias toward tabular data?
*cite in Overview paper: syntactic data transformations (formatting): "//parsing or reformatting data to ensure they can be read.//"
*"visualizing raw data" won't work for large text document collections when diagnosing data problems. data transformation must come first, using tools like Google Refine; what is an appropriate summary visualization for a text document collection? Pixel-oriented?
*duplicate record detection relevant to Overview, as is domain-specific data types.
*//"Often this process requires writing idiosyncratic scripts in programming languages such as Python, Perl, and R"//: scripts, if documented and commented, are at least more reproducible and transparent than a set of disjoint interactive tools or widgets that wrangle data. A single interactive workflow / tool with auditing / provenance tracking would be required to reach the same level of transparency / reproduciblibility.
!References
<<bibliography>>
!![Saraiya2004], [Saraiya2005] - ~Insight-Based Evaluation
<<cite Saraiya2004 bibliography:Bibliography>>, <<cite Saraiya2005b>> conducted an evaluation of 5 visualization tools used for analyzing microarray datasets. Due to the exploratory nature of the analysis of these types of data, their evaluation focused on measuring insight gained during use of the tool. Their [[Insight-Based Evaluation]] protocol required defining what is meant by ''insight'' in this context, which included the development of new domain-relevant hypotheses.
''Methodology'': 3 datasets and 5 visualization tools (//Clusterview//, //~TimeSearcher//, //HCE//, //Spotfire//, and //~GeneSpring//) were used in the study. Among the tools a variety of visual representations and interactive techniques were used. 30 participants with domain knowledge were recruited, falling into 3 categories (domain novice, expert, and domain software developer). The method combined controlled experiment and usability testing (i.e. the [[Think Aloud Protocol]] was used). Six participants were allotted to each tool, and 2 datasets were allotted to each tool.
''Measures'': After a tutorial using the tool and during the analysis, participants were queried at intervals about the amount insight gained. ''Insight'' was defined as a unit of discovery, an individual observation about the data. Along with a domain expert, insights were scored, characterized by the fact discovered, the time to reach the insight, the domain value of the insight, whether the insight led to the formation of a new domain-relevant hypothesis, whether the insight pertained to breadth or depth, whether it was the result of directed inquiry or it was a serendipitous discovery, its correctness, and the domain-relevant category it fell under (which were not predefined, determined after the data was collected). The total domain value of all insights and their categories were primary dependent measures.
Other dependent measures included the participants' initial questions about the dataset, their time spent with the tool, the time until the first insight, the total amount learned (as a percentage, periodic and final) - as perceived by the participants (which included estimation of how much the tool was //not// allowing them to discover, which visualization techniques were used, usability issues encountered, and their background / demographic information.
''Results'': Better results were achieved for a subset of dataset and tool pairings, suggestive of certain tools' strengths for analyzing certain types of datasets. Most insight gained was //breadth//, rather than //depth//. Few hypotheses were generated with any of the tools in the experiment. The total number of insights did not depend on participants' background (however novices needed more prompting and were less confident in their findings).
!!!!Comments & Questions
*Evaluating exploratory analysis appears to be largely qualitative and requires the presence of a domain expert for coding insights. Can insight be defined agnostic of the domain? Can it be quantified for quantitative analysis and meta-analysis?
*Participants varied in terms of domain expertise (albeit all had a baseline cutoff). They also varied in their day-to-day roles (software developer vs. senior researcher vs. junior research assistant). However the tools did not leverage domain expertise well. Does this have to to with the session length, the datasets used?
*A 15-minute tutorial with the tool was the extent to which the participant learned the tool prior to the evaluation. The authors justify this that success in the initial usage period is critical for tool adoption. They later admit that there was a lack of sufficient training. Regardless, they stress the need for longitudinal [[Insight-Based Evaluation]] with domain experts working with their own datasets.
**Related to the previous point, the authors point out the lack of motivation in a short controlled study such as this. Motivation for gaining insight in [[Insight-Based Evaluation]] is not as straightforward as motivation for reducing errors, increasing accuracy, lowering completion time in conventional user studies.
**Longitudinal [[Insight-Based Evaluation]] with domain experts working with their own datasets may additionally solve the motivation problem, however quantitative analysis of the results in a longitudinal setting across many participants may no longer be possible.
**Similarly, generating new hypotheses is also related to motivation and session length.
**A balance of depth and breadth insights may also be achieved with longer session lengths and personally-relevant datasets.
*Counter-balancing was not attempted due to the availability of similarly controlled test datasets and learning effects. The type of dataset and tool were confounding factors, but the authors stress that quantitative comparisons between the tools were not a focus of the research.
*"Usability can outweigh the choice of visual representation": a tangential discussion statement - not the focus of the paper but mentioned based on anecdotal reports from the participants.
!![Saraiya2006] - Longitudinal ~Insight-Based Evaluation
In a follow-up paper to their [[Insight-Based Evaluation]] in <<cite Saraiya2004>>, <<cite Saraiya2006>> conducted a longitudinal self-reporting study of visual analysis. While still having an emphasis on the number and type of insights generated while using a visualization tools, the study focused on a smaller set o motivated, domain embedded representative users (bioinformaticians) engaging in their own research with their own data over the course of three months in the course of their daily work (rather than an observational lab study of novice (but domain-knowledgeable) users with limited training). Furthermore, the current study involved analysis of diaries to record insights, rather than an obtrusive observational method, following domain experts from the point of learning a tool to the point of generating many useful insights via exploratory analysis. Therefore the study protocol was easy integrated into their daily work and they were permitted to use a variety of available visualization tools and other processes (including calling tech support and reading help manuals).
Participants used a combination of new tools and software already familiar to them (i.e. Excel), which may be inefficient but the cost of switching tools was seen as too high. Their process was captured and compared positively to existing task frameworks (i.e. overview, zoom, and filter). Dynamic queries and transparent filtering techniques, along with simple visual representations were preferred. Insights were generated and validated by domain expertise. Most exciting insights came after 1.5 months of analysis and several months of learning time. There was nevertheless much back-and-forth processing between several visualization tools.
The researchers also comment on the effectiveness of the longitudinal diary technique, which reduced the amount of data (i.e. observational data), and reduced the requirements of the study for both participants and evaluators. There was still an abundance of rich data (learning, insight generation) from diaries (screen shots, noted insights, interviews). A better coding of the analytic process is needed in the future.
!!!!Comments & Questions
*An appropriate follow-up to their earlier work, however dropping metrics for scoring and labeling insights generated.
*Only 2 representative users involved. A lot of data is missed when not observing (written accounts are subjective and likely leave a lot unsaid).
!![Saraiya2010] - ~Insight-Based vs. Benchmark Task Evaluation
<<cite Saraiya2010>>'s BELIV '10 paper, comparing standard task-driven quantitative evaluation against the qualitative [[Insight-Based Evaluation]] method from earlier papers. They observe that the undirected tasks user perform in the [[Insight-Based Evaluation]] account/correspond to the directed tasks in the benchmark study, with additional unforeseen tasks added. They also reach the same conclusions in the [[Insight-Based Evaluation]] and gain additional insight, when compared to the benchmark method.
''Methodology'': 60 participants, 10 x 3 vis alternatives x 2 methods. Brief introduction to the visualization followed by experimental protocol (benchmark task method with 7 tasks, time and correctness dependent measures, OR [[Insight-Based Evaluation]] with [[Think Aloud Protocol]].
''Results'': [[Insight-Based Evaluation]] took longer. Insights were very similar to correct responses to tasks in the benchmark task method, or they were amplified / similarly confirmed. Choosing predefined tasks is likely to force participants to conform to a certain line of thinking, performing analyses they would not otherwise perform. The benchmark task method appears to be more perceptually oriented. Insights in this limited study were of little value, ranked by a domain expert.
''Discussion'': Data analyses is more difficult and subjective with the [[Insight-Based Evaluation]], and results had higher variance (affected more by individual differences). The effort is offset by the time and expertise saved in the design phase by not requiring to design predefined tasks (a nontrivial job). Unmotivated subjects are easier to spot in the [[Insight-Based Evaluation]]. The [[Insight-Based Evaluation]] can also be applied in longitudinal settings. [[Insight-Based Evaluation]] provides the possibility of finding new task types from users.
!!!!Comments & Questions
*I was surprised to see them bring the [[Insight-Based Evaluation]] back into a lab setting with unrepresentative novice users and data sets. This seems like a step backwards - was it really necessary to hit the final nail into the benchmark task method's coffin?
*This methodology also revives the motivation and learning critiques from their earlier laboratory study. The insights generated were also superficial and could not be easily ranked by domain importance by an expert.
*Their findings questions validity of many benchmark task-based evaluations proposed in the literature. However tasks vary in the amount of domain expertise or oversight that contributed to their selection - some tasks may be more ecologically valid than others.
*Finding //relevant// new task types from [[Insight-Based Evaluation]] may only be possible with appropriate domain users in a longitudinal study.
!![North2011] - On Characterizing Insight
<<cite North2011>> is the follow up IV journal paper to the <<cite Saraiya2010>> BELIV workshop paper, expanding on the differences between the task-based and insight-based evaluation methodologies for comparing visualization alternatives.
The insight method treats tasks as dependent measures. It assess how a visualization system //promotes// tasks, rather than how it //supports// tasks, which is a question better suited for the task-based method. The insight-based evaluation can address higher-level questions regarding task taxonomies, conclusions about visualizations, time spent by participants in the study, effort spent analyzing the data. This is particularly effective when the analysis task is exploratory in nature.
''Methodology'': see <<cite Saraiya2010>> above. Analysis of the insight-based data was categorical, not quantitative. Categories were developed in an open-coding fashion, and in discussion with domain experts. However, in future work the authors entertain the possibility of categorizing insights using the task taxonomy of <<cite Amar2005>> (low-level components). Domain expert scoring wasn't effective since the study was not longitudinal and most insights were surface-level (dry data analysis, without much biological inference), very little analytical depth, and thus there was very little variation in correctness, breadth/depth. This is despite participants having domain knowledge. The small toy-zised data set was another issue that likely contributed to this problem. A longitudinal study with a full data set may provide more variation in these respects. They quantified insights-per-minute, something that couldn't be done in a longitudinal setting. They state that enforcing a fixed time on an insight study could greatly bias the results (what about longitudinal insight studies?).
The insight-based evaluation can also gauge motivation and user engagement, but this may require constant observation. It can also elicit feedback regarding usability and visual encodings used.
The authors acknowledge that some individuals are more communicative than others, some insights may seem trivial to some but not to others. Some participants thought there was a 'catch' to the open-ended insight study - they believed to be missing something obvious.
!!!!Comments & Questions
*Open-coding the insights into categories - could this expand into a full-blown [[Grounded Theory]] analysis, without a real need to enlist domain experts to verify/generate insight categories?
*A within-subjects study - only one method and interface experienced by any single participant (time constraints, undergraduate research subjects). For a longitudinal study, this is no longer much of an issue, if participants experience multiple interfaces / systems / techniques, with different datasets, order counter-balanced across participants.
*RE: insights-per-minute. is this really all that informative? Wouldn't insights generated before a longitudinal deadline be more telling? Leave time up to the research participants.
*Participants need to be encouraged that no insight is too trivial, they should't be suspicious of researchers' intents - there is no 'catch' to the study
*Observation and longitudinal insight - how to compromise? Nagging reminder emails? Can you still collect information on engagement and motivation?
!![O'Brien 2010] - Applying an ~Insight-Based Eval. in Bioinformatics
<<cite O'Brien2010>> performed a mixed method evaluation of a bioinformatics visualization application called Gremlin. They gathered qualitative feedback and quantitative results, the latter using a lab-based [[Insight-Based Evaluation]], comparing Gremlin to Circos, the state-of the art.
They quantified total insights, hypothesis-driving insights (determined by a domain expert), insights-per-minute, and insight complexity (determined by a domain expert): A. simple observations, B. detailed observations, and C. detailed observations with context (cross-referencing).
Gremlin outperformed Circos on all metrics.
>//while we concede visual perception is integral to insight generation, we posit that flexible interactive interfaces within a visualization are paramount with respect to sparking insight//
!!!!Comments & Questions
*Types of insight were determined a priori.
*They had only 5 subjects, no full counter-balancing. They experienced both types of visualization (4 pairs, alternating between visualization types)
*Subjects had to list expected insights beforehand, then return to the list after the session to 'check off' each expected insight if they discovered this during the session; this turns their methodology into something akin to a task-based evaluation - how can subjects know what they are expected to find, or what tasks a system will support, a priori. Gremlin outperformed Circos in this regard
*A strong preference for Gremlin - surprising, subjects were lab mates / friends of authors
*Study proceeded until no more insights could be generated; sessions were reviewed later over video with domain expert;
*What's the difference between generating and sparking insights?
!![Shneiderman2006] - ~MILCs (Multi-dimensional In-depth Long-term Case Studies)
<<cite Shneiderman2006>> have proposed [[MILCs]] (Multi-dimensional In-depth Long-term Case Studies) for studying expert users of [[Information Visualization]] tools working on their own problems, in their normal environments. [[MILCs]] have been applied previously in HCI settings; the authors' current work is to refine this method and expand its scope to cover [[Information Visualization]]. There is a growing desire for alternative evaluation methods in the field, particularly when considering the creative nature of tasks users undertake while using [[Information Visualization]] tools: the need to make and refine hypotheses, look for patterns, produce insights, and discover through innovation.
In [[Information Visualization]] settings, the outcome of [[MILCs]] could lead to the refinement of a tool or an understanding of design guidelines, or the achievement of expert users' goals through use of the tool. In the example used by the authors, a researcher may have recently developed a novel approach and seeks to evaluate it in a real-world setting. Alternatively, a larger organization may want to deploy an existing tool, and thus wants to evaluate its effectiveness in different settings.They recommend spending at a minimum several weeks with 3-5 domain experts, with the following considerations:
*specify focused research questions, goals
*identify 3-5 appropriate domain expert users
*document their current work practices and tools
*record the goals of the domain experts
*schedule observation and interviews
*instrument your tool to record usage data
*provide a log book for your domain experts
*provide training on existing or new tools
*conduct visits and interviews
*encourage best practices for each task, regardless if whether it involves the tool being evaluated
*modify the tool be evaluated as needed
*document successes and failures
The authors plan in the future to study the use of this evaluation technique in collaborative settings with teams of domain experts, thus necessitating larger research teams over longer periods of time.
!!!!Comments & Questions
*There is little discussion of design stages or scenarios, with regards to when [[MILCs]] should be considered.
*A very readable and satisfying historical review of evaluation methods dating back 400 years.
*The authors specify the need for teams of 3-10 researchers, but do not divulge predefined roles or prerequisite knowledge for team members.
*While the position of the paper calls for new evaluation methods to be used in the field, it is implicitly inferred that [[MILCs]] occur after user performance testing and usability evaluation.
*''Analysis'': how do you analyze your results - can they be quantified? Is this done on a case-by-case basis?
*<<cite Tory2008>> sez: <<cite Shneiderman2006>> provides "many suggestions for conducting long-term field studies, but limited guidance on how to analyze the resulting data.
*<<cite Valiati2008>> sez [[MILCs]] have now been used in 7 studies (as of 2008).
!![Perer2009] - MILC follow-up
<<cite Perer2009>> conducted several case studies of domain experts conducting visual analysis using a visualization tool with integrated statistical functionality, furthering and building on the methodology of [[MILCs]] from <<cite Shneiderman2006>>. Their case studies result in design guidelines (alternatively these could be used in [[Heuristic Evaluation]], although they are not explicitly posed as such). Furthermore, they present a task taxonomy for visual analysis with the aid of statistical functions for computing variables. The task taxonomy can be reconsidered as design heuristics.
''Methodology'': interview (1 hour), training (2 hours), early use (2-4 weeks, make software changes on request, weekly phone and in-situ interviews), mature use (no additional changes, weekly telephone and in-situ interviews), outcome interview (1 hour) - shared notes and screenshots from participants collected. No automatic software logging / recording.
''Design guidelines'': a task taxonomy for dealing with inherent and statistically relevant computed attributes of the data:
*''reconfigure'' - provide statistical perspectives, rather than reconfigure inherent attributes into alternate visualizations, reconfigure current visualization based on computed attributes
*''connect'' - coordinate stats and vis (i.e. brushing)
*''encode'' - represent computed variables to augment vis.
*''selection'' - persistently marking interesting computed variables
*''filter'' - filtering by computed variables rather than by removing data points (i.e. outliers)
*''abstract / elaborate'' - reduce/increase detail (i.e. clustering, de-clustering)
*''explore'' - reach insights, suggest places to explore (personalization, support different styles of interaction, systematic yet flexible
!!!!Comments & Questions
*Methodologically, there is not more in this paper than in the earlier MILC paper. However the added value comes with the design guidelines, if considered as heuristics.
*Combines meta-analsyis in the side panels, farming the difficulty and need for evaluation in the field, similar to <<cite Lam2011>> in which several ~InfoVis publishing venues are surveyed for recent published works containing evaluation.
!![Valiati2008] - ~MILCs across domains
<<cite Valiati2008>>'s BELIV '08 paper applied the [[MILC|MILCs]] methodology in three different domains. Their goal was to study and compare the outcomes of difference case studies: user behavious, tasks, usability issues and their impact. They recruited motivated users from 3 domains with their own data sets and existing visualization tools. Results were classified into four categories: ''high-level analytical questions'', ''discoveries and insights'' resulting from the data analysis, ''usability'' problems, and different occurrences of ''tasks'' and subtasks described in a visualization task taxonomies, which is reported in their earlier BELIV paper, <<cite Valiati2006>>. Their observations are also in line with <<cite Gonzalez2003>>, in that their users used the tools in a complimentary manner, and sparingly between observation sessions.
!!!!Comments & Questions
*How did they analyze their results?
*No substantial commentary as to whether [[MILCs]] work? The answer appears to be "maybe". Benefits remain unclear (listed as future work).
*They conducted three [[MILCs]] in different fields. Can they generalize? They don't address this adequately. They state in the related work section that multiple domains allows you to generalize. Did they believe this after the study?
*They did not follow the [[|MILC|MILCs]] methodology 100% (they did not log user actions with software tools). They followed the [[MILC|MILCs]] methodology to an extent, but did not discuss how data was analyzed, they just present 3 case studies, broken down into the user's goals, their data set, and their techniques/tools (a breakdown I appreciated).
*Their literature review is just a summary, no critique is offered.
*re: their results - how did they classify the data into four categories?
*Big claim in section 4: "users always have a purpose in mind when doing data analysis, knowing a priori which hypothesis they will check." TM comment: open exploration vs. hypothesis confirmation?
*What were their limitations (section 5)?
!![Amar2004] - Analytical Gaps (~High-Level Task Taxonomy)
<<cite Amar2004a>> discuss [[Analytic Gaps]], barriers to higher-level analytic tasks (decision making, learning) when using [[Information Visualization]] tools. These gaps can be differentiated between a ''rationale gap'', between perceiving a relationship and appreciating the utility and confidence of the relationship, and a ''worldview gap'', between what is shown to a user and what is hidden from the user, the amount of information needed to make a conclusion and decision based on the data.
The authors propose an systematic valuation method akin to [[Heuristic Evaluation]] by expert evaluators, identifying analytic gaps to assess whether the tool supports higher-level cognitive analytic tasks.
Systems designed and evaluated with consideration of [[Analytic Gaps]] will focus on ''analytic primacy'', supporting high-level tasks, rather than ''representational primacy'', which focuses on low-level tasks. However, a balance must be struck such that the tool is not a //black box//, in that the underlying data can still be viewed and interpreted: user control must be provided.
A taxonomy of analytic knowledge tasks (decision making under uncertainty, learning a domain) is provided for conducting the evaluation:
*Rationale-based tasks
**expose uncertainty (measures and aggregations, effect of uncertainty on outcomes)
**concretize relationships (what comprises the representation of a relationship and its outcomes)
**formulate cause and effect (clarify possible courses of causation)
*Worldview-based tasks
**determine domain parameters (provide transfer/acquisition of knowledge or metadata about domain parameters within a data set)
**multivariate explanation (allow discovery of correlation and constraint)
**confirm hypotheses (allow formulation and verification of hypotheses)
The authors conclude the paper by applying these tasks in both a design and evaluation setting.
!!!!Comments & Questions
*re: section 1.1: Decision makers rely more on the the macro-level for making decisions. Breadth rather than depth? This seems at odds with the idea that breadth-level insight is shallow and not as valuable as depth-level insight (<<cite Saraiya2004>>).
*re: section 1.2: Does each new scientific domain require a new visualization?
*re: section 1.3: claim that most systems do not represent ''uncertainty'', ''cause and effect'' particularly well.
*No mention of whether evaluation of analytic gaps requires domain expertise, other logistical constraints.
*If another dimension of the taxonomy is decision-making vs. learning - where do the knowledge tasks fall?
*<<cite Saraiya2004>>'s notion of ''insight'' is related to the high-level knowledge tasks referred to here. <<cite Amar2004a>> use the taxonomy of knowledge tasks to explicitly define what insight is needed for.
**The former's [[Insight-Based Evaluation]], with a domain expert, can be analyzed quantitatively, but without regard to specific knowledge tasks.
**<<cite Amar2004a>>'s taxonomy does not place different values on knowledge tasks and thus evaluations are qualitative and results must be analyzed on a case-by-case basis (as some domains will place different values on the different knowledge tasks).
**Could a joint evaluation framework be devised that combines the taxonomy of knowledge tasks with [[Insight-Based Evaluation]]?
*There exists a later journal paper, however it does not contribute enough additional material to make it worth reading.
!![Gonzalez2003] - ~Post-Deployment Eval.
<<cite Gonzalez2003>> studied the creative discovery process using an information visualization tool in a long-term qualitative field study among administrative data analysts. The analysts, accustomed to working with large amounts of administrative data in tables and spreadsheets, were trained with the //~InfoZoom// visualization tool, and were instructed to make use of the tool with their own data as part of their daily workflow.
''Methodology'': The study began with a initial semi-structured interview lasting 50min to understand participants' job descriptions and workflows. Participants were given a tutorial with //~InfoZoom//. The authors interviewed the participants once a week over the course of 6 weeks; during the interim time participants used //~InfoZoom// with their own data. A final interview regarding the tool was conducted to conclude the study.
''Results'': Using //~InfoZoom// was helpful when typical work practices broke down, when atypical information analysis requests were made of the participants. Participants found the tool to flexible (allowing filtering, focus, and comparison), that it provided a full overview of their data, and that it enabled extraction of charts and reports, which fit naturally into the analysts' workflow. Participants found it difficult to use to the tool to complement existing work practices for typical analysis tasks, which were already well-refined. However, //~InfoZoom// allowed for creative exploratory analysis, outside of the scope of their typical analysis, whenever participants wanted to verify or debunk a hypothesis, assumption, or suspicion, usually tangential to their own work. Creative analysis became a motivation for participants to use the tool. Finally, the tool became part of the process of creatively pre-exploring the data before applying more rigorous statistical techniques: it became part of their work practices as a preliminary, rather than complimentary process. Those who wanted to use //~InfoZoom// to complement their other tools (involving compatibility) and work practices were inconvenienced.
!!!!Comments & Questions
*The authors suspected introduction of the tool into the environment would result in a new data analysis work practice. Perhaps, like the participants in early stages of the study, they imagined visualization use to be complimentary to existing practices. In the end, the use of the visualization tool took on a preliminary role, rather than complimentary - a visual pre-analysis. Participants were surprised by this, but it's hard to tell if the authors were. It appears that the tool was flexible enough to take on this role. However, a pre-analysis may not employ the full capabilities of the tool, which may result in a resistance to tool adoption based on cost.
*Were //~InfoZoom// to be compatible with the analysts' existing software tools - would the results have been different?
*They do not report how long the participants spent with the tool - were they instructed to use it as much as possible? only when convenient and compatible with their work practice? only when they had spare time? or were they instructed to use the best practice for the job at hand, regardless of whether or not it involved //~InfoZoom//?
*They did not log software use (at least they didn't report it), nor did they report or ask about usability issues (other than cross-tool compatibility).
*Creative discovery processes, while not structured, seem to be secondary tasks to these analysts - only needed in rare cases or when they had spare time. While interesting and motivating, it doesn't appear to be a big part of their job, which are implicitly structured (or at least they have imposed structure through the use of templates).
!![Gonzalez2003a] - ~Post-Deployment Eva (Follow Up)
<<cite Gonzalez2003a>> expands on the above work in this paper. They relate the story of deploying visualization tools to individuals who don't need vis for most purposes, who don't have time for vis. Formatting and reconfiguring data between //~InfoZoom// and their existing tools becomes too much of a hassle. The subjects never replace their existing workflow, but rather only occasionally complement their routines with use of //~InfoZoom//.
!!!!Comments & Questions
*This paper doesn't contribute new material compared to their other related work (they attempt the above study with several additional subjects, to no greater amount of success).
*It's also obvious that //~InfoZoom// is the wrong tool to deploy (a novel technique without support for integration with existing software).
*Less methodological details are provide here compared to <<cite Gonzalez2003>>.
!![Isenberg2008] - Grounded Eval.
<<cite Isenberg2008>> discuss a [[Grounded Theory|Grounded Evaluation]] approach to exploratory, pre-design evaluation. They advocate pre-design [[Grounded Evaluation]], as opposed to starting with a design followed by after-implementation evaluation, as by this point neither the design nor the evaluation has been grounded in the context in which the tool has been deployed. After-implementation is still necessary, however it is best if this evaluation is grounded in the context in which representative users use the tool: appropriately complex datasets and tasks, realistic stress levels and data exploration experience, distractions, cognitive processing capabilities, and concurrent and existing data analysis processes occurring in the environment.
They also advocate the need and strengths of qualitative evaluation and rich data collection, and the differences in how this data can be formally coded (i.e. data driven / [[Open Coding]], driven by previous research, theory-driven). This process will produce themes for presenting a consistent picture of the context.
The authors illustrate the success of this process with three case studies. They conclude the paper with a discussion on how the larger research community could make efforts to embrace this exploratory, grounded pre-design evaluation.
!!!!Comments & Questions
*Repetitive, but drives the point home. A Somewhat unnecessary argument if the reader isn't coming from the ~InfoVis community, but rather the HCI community.
!![Trafton2000] - Cognitive Task Analysis
<<cite Trafton2000>> conducted a [[Protocol Analysis]] at different stages of expert meteorologists' workflow, according to a previously conducted [[Cognitive Task Analysis]], corresponding to different stages of producing a weather forecast.
''Method'': the [[Observational Study]] depicted, including observations by domain experts, a recording of utterances using the [[Think Aloud Protocol]], the use of video recording, and a protocol analysis tool (//~MacSHAPA//), allowed researchers to code the interactions along several dimensions:
*visualization type in use: picture, chart, graph, text
*utterances / written statements reflecting visualization usage: goal statement, information extracted or inference made, brief-writing
**secondary categorizations: qualitative vs. quantitative, integrated vs. non-integrated (multiple visualization sources), source (visualization or QMM: qualitative mental model)
''Results'': the researchers documented the meteorologists' protocol based on their [[Protocol Analysis]]. The expert users first attained a qualitative big-picture view of the data with no explicit detailed data extraction (qualitative or quantitative). The researchers observed how qualitative mental models were constructed, verified, and adjusted based on the integration of several complex visualizations, and how quantitative information was extracted from their qualitative mental models at the brief-writing stage, which relied heavily on domain expertise to convert qualitative knowledge into quantitative information. They proposed that expert users of complex visualizations use heuristics to deal with the large amount of data they encounter: data extraction was integrated across several sources, it was predominantly qualitative and directed by a goal, rather than quantitative and serendipitous.
They speculate that the qualitative mental model is imagerial and spatial, subject to alignment, metric, and rotation errors. They believe their findings generalize to many domains in which expert users must make a prediction based on an integration of many data sources.
!!!!Comments & Questions
*An example of a qualitative lab study with realistic data, representative users, and an observational method
*task analysis drove the protocol coding (i.e. not open coding, but research-driven coding)
**The [[Cognitive Task Analysis]] and its results weren't discussed at length - the task stages and the evidence for them was agreed upon by representative users and domain experts, but how it was conducted and how its results were analyzed were not discussed.
*Section 4.3 - practical implications: read: design implications - a shorter section than anticipated considering the publishing venue - suggesting that systems support the comparison and integration between different visualizations, and that an intelligent agent could retrieve secondary sources of information while a related visualization is being used for extraction.
*Given the date, this is likely one of the first qualitative observational studies conducted for analyzing the information retrieval and integration process. It corresponds with <<cite Lam2011>>'s first scenario of trying to understand the user's workflow (regardless of the presence of a visualization tool or tools).
!![Seo2006] - Longitudinal Case Study Eval.
<<cite Seo2006>> evaluated a new ranking feature in a popular visualization tool (//HCE//) used predominantly by microbiologists (however many other fields also make use of it). Their evaluation consisted of three longitudinal [[Case Studies]] performed with three domain experts in three different domains. This was supplemented by an email [[User Survey|Questionnaires]] with 57 respondents, again from a variety of fields. Subsequent analysis allowed the researchers to determine the efficacy of the new feature in the application, how it was being used, and how it could be improved in later development. Additionally, they sought to validate their guiding principles for building the new feature, used for analyzing multivariate data sets. This was based on their //GRID// (Graphics, Ranking, Interaction for Discovery) workflow premise given a clear goal: (1) study variables in 1D, (2) study combinations of variables in (2D), (3) find features, (4) rank by these features to guide insight, and finally (5) confirm via statistical analysis. This workflow replaces opportunistic / serendipitous discovery with a repeatable, orderly, and thorough process.
As opposed to controlled research studies conducted in lab settings, these case studies were conducted over the course of 6 weeks in which participants were encouraged to use the tool with their own data, working on their own problems. The researchers acknowledges that exploratory data analysis can take days or weeks and time must be allotted to learning the tool and integrating it into one's workflow.
''Method'': 6 weeks of weekly participatory observations and interviews lasting 30min or longer. Participants were to write a short report based on their experience after 6 weeks. Observations and interviews had specific questions in mind regarding the tool and particularly its new feature, rather than a [[Grounded Evaluation]] making use of [[Open Coding]]. A user survey with 57 respondents supported the three case studies.
''Results'': the researchers' //GRID// principles were validated, and an understanding of how GRID is performed in context will guide their subsequent design. They also encourage other researchers to adopt this evaluation approach.
!!!!Comments & Questions
*The authors stress the importance of conducting case studies in multiple domains. This tool is exceptional in that it is general-purpose. With many other tools this is not possible and we must sacrifice generalizability for satisfying a smaller set of targeted domain users, and thus these case studies may only be valuable on a case-by-case basis. Nevertheless, if the tool is not domain-specific, efforts should be made to include multiple domains in one's evaluation.
*Case study observations in too much detail for my purpose of understanding the researchers' process.
*They stressed that developers of visualization tools must spend ample time trying to understand representative data sets. This seems obvious.
*<<cite Ellis2006>> might argue that their targeted questions asked during observations and interviews would not constitute novel research but rather effective responsible product design and iteration. This study is very much a post-design qualitative piece in which the researchers sought to validate a tool and the guiding GRID principle which inspired it, rather than understand for who, when, and how this and other principles apply and interact in context.
!![Scholtz2010] - Eval. Heuristics from VAST contest winners
<<cite Scholtz2010>> conducted a meta-review of VAST challenge entry reviews to lay the groundwork for the creation of qualitative [[heuristics for evaluating|Heuristic Evaluation]] Information Visualizations (likely to appear in subsequent journal paper from IV Jul '11). VAST challenge entry reviews incorporated the opinions of visualization experts and visual analysts, which addressed not only the visualization itself, but also the analysis process (incorporating both categories of evaluation scenarios from <<cite Lam2011>>). Therefore these forthcoming evaluation heuristics should be part of [[Expert Evaluation]]. An agreed-upon final set of heuristics remains to be determined.
Currently, reviewers evaluate challenge entries quantitatively and qualitatively. The latter encompasses usefulness, efficiency, and intuitiveness. The author re-wrote many reviewers' comments regarding these criteria as groundwork for qualitative heuristics for evaluating the analytic process, visualizations, and interactions. (see Table 1 for full list). They noticed that visualization experts paid more attention to all three, while visual analysis experts concentrated on the analytic process.
!!!!Comments & Questions
*It is not clear what type of expert is needed for the [[Expert Evaluation]] (visualization experts and visual analysts, or usability experts?). This remains an open question.
*Another focus of the paper asks what materials should be made available to VAST challenge reviewers, and what aspects
*See journal paper for heuristics?
* Table 2 hard to interpret - consider visualization!
!![Isenberg2008a] - Eval. of Collab. Visual Analysis
<<cite Isenberg2008a>> conducted an observational laboratory study of individual and collaborative visual information analysis. This study resulted in design implications / heuristics for designing and evaluating Information Visualization tools. General processed were studied rather than low-level perceptual or interactive tasks. The observation study did not involve an existing visualization tool, but shared printed materials (graphs and charts of toy data sets). They sought to understand these processes rather than interactions with existing tools.
''Methodology'': a brief tutorial of the research materials and tasks were provided. Individuals were instructed to use the [[Talk Aloud Protocol]]. Group discussion was recorded and analyzed similarly. They were instructed to approach the tasks however they saw fit.
''Findings'': the researchers' observations revealed that information analysis is not temporally organized in consistent ways within or between groups, but that individual sub-processes were discernible: browse, parse problem (if one is defined), discuss collaboration style (complete task division, independent and parallel, joint work), establish task strategy, clarify extracted information, select / extract, operate (most time-consuming - higher-level cognitive work on the data), validate / confirm solution. Groups were slower but more accurate.
Comparisons are made with the [[Information Foraging and Sensemaking]] cycle and models of collaborative analysis. They found their findings to fit well with these models, however discussion of collaboration style was not explicitly represented in these other models.
''Implications'': The analyses processes were temporally flexible, as was collaboration style. Designs should support this criteria, and personal/group workspace organization. Do not assume a standardized temporal flow of one sub-process to another.
!!!!Comments & Questions
*Focused and open-ended analysis problems were not specific to a domain. Non-domain users with no specialized knowledge ( i.e. undergraduate students). Data sets were SPSS demo materials. Collaborative materials were printed from SPSS. Difficult to generalize results (surprising as this is a CHI paper). 4 groups of singles, pairs, and triples (N = 24).
!![Mayr2010] - Characterizing Problem Solving
<<cite Mayr2010>>'s BELIV '10 paper argues that analysis of representative users' creative problem solving techniques sheds more light on the exploratory visual analytics process than [[Insight-Based Evaluation]], and is thus a superior evaluation technique. They define types of problems (well-defined (one solution), ill-defined (exploratory)), as well as problem solving strategies (schema-driven, search-based). They encourage visualization designers to accommodate both types of strategies.
''Methodology'': real-world data sets of varying levels of real-world complexity, two visualization systems. Interaction was logged, tracked viewing behaviour, and used the [[Think Aloud Protocol]]. 12 participants had to solve a set of well-defined and ill-defined problems. Tasks included //reading the data// (locating/extracting data, well-defined), //reading between the data// (relationships, interpolation, well-defined), and //reading beyond the data// (extrapolation, ill-defined).
''Results'': strategies were highly variable within and between subjects for tasks and tools, but some data sets suggested certain strategies (an interaction of data set and strategy). A correlation between strategy selected and solution quality was found, but nevertheless multiple strategies should be afforded in interfaces. Their approach was easy to apply and replicate.
!!!!Comments & Questions
*Their methodology and relations to other ongoing projects were unclear.
*It wasn't stated how their problem sets (both well-defined or ill-defined) were selected or who generated them (and how much time this took relative to the analysis of their results).
*I wasn't buying their conclusion that analyzing problem solving techniques answers more than [[Insight-Based Evaluation]]. They do not relate solution quality to insights (they assume that the problem solving process leads to insight generation).
**They could have done a head-to-head comparison of the two techniques (albeit limited being in a lab setting with artificial problem sets, users, and data sets), with dependent variables being time to analyze results vs. time to design predefined problems, amount of expert / domain knowledge required, and when expertise is needed.
*They admit that problem-solving subprocesses are still unclear, despite a wealth of literature on low-level and perceptual-level task taxonomies relating to problem solving, insight generation, information foraging.
*Their method did not compare head-to-head against the [[Insight-Based Evaluation]], so claims about which is more effective are questionable.
*Did not appreciate writing style. I will forego reading the journal paper, but a related IEEE journal paper re: participatory design looks like it may be relevant.
!![Valiati2006] - High- and ~Low-Level Task Taxonomies for Eval.
<<cite Valiati2006>>'s BELIV '06 provides a taxonomy of high- and low-level tasks to be used in evaluation. The taxonomy borrows heavily from other related work, which they survey in detail.
Their taxonomy integrates high- and low-level tasks, as well as analytic, cognitive, and operational tasks A hierarchy cannot be established between them (nor a workflow?):
*''identify'' - clusters, correlations, categories, properties, patterns, characteristics, thresholds, similarities, differences, dependencies, independencies, uncertainties, variations
*''determine'' - mean, median, variance, standard deviation, amplitude, percentile, sum, proportions, differences, correlation coefficients, probabilities, other statistics
*''visualize'' - //n// dimensions, //n// items, data, domain parameters / attribute information / metadata
*''compare'' - dimensions, items, data, values, clusters, properties, proportions, positions / locations, distances, graphical primitives
*''infer'' - hypotheses, rules, trends, probabilities, cause / effect
*''configure'' - normalization, classification, filtering, zoom, dimensions order, derived attributes, graphical primitives
*''locate'' - items, data, values, clusters, properties, position / locations, distances, graphical primitives
''Methodology'':They conducted a small evaluation on 2 visualization tools constructed using the ~InfoVis toolkit (//Parallel Coordinates//, //~RadViz//). A small number of study participants recruited from the computer science department (and one domain expert, a biologist) were recruited. A set of four directed questions were used to assess the validity of the taxonomy. They use the benchmark cars data set.
!!!!Comments & Questions
*A long related work, given the total article length, a good job relating to previous taxonomies (the novelty of this task taxonomy isn't very apparent - is it because it doesn't distinguish between high- and low-level tasks, as well as analytic, cognitive, and operational tasks, between orders of tasks, hierarchies, dependencies?)
*Their taxonomy is the main contribution, preceded by their decent lit review. Their own study is not compelling enough, were the problem sets chosen knowing that they would easily validate their taxonomy? What is the breadth of these problems? Also, the benchmark cars data was used - how generalizable are these findings with novice users and a simple toy data set and only 2 forms of visualizations? What was the training procedure?
**I appreciated how their didn't specify dependencies or progressions of tasks, or lengths of tasks, suggesting that tasks can very in length and when they occur in a sequence
!![Zuk2006] - Heuristics for Eval.
<<cite Zuk2006>>'s BELIV '06 paper uses three previously published sets of heuristics for evaluating Information Visualization tools in a comparative [[Heuristic Evaluation]] of a visualization tool. They discuss the value of each set of heuristics and where and when they ought to be applied.
They realize that the three sets of heuristics are complementary, as Zuk & Carpendale's heuristics pertain to the low-level sub-task scale (perceptual and cognitive heuristics), while Shneiderman's [[Visual Information-Seeking Mantra]] (overview first, zoom and filter, relate, extract, history - details on demand) pertains to the task level (exposing usability problems), and <<cite Amar2004a>>'s set of heuristics (based on their [[Analytic Gaps]]) relate to high-level analytic tasks (specific to the domain, decided at the scope level of the project) (exposing uncertainty, concretizing relationships, determining domain parameters, providing multivariate explanations, formulating cause and effect, and confirming hypotheses). Therefore there exists considerable overlap when these heuristics are applied. As a result, the authors are unable to taxonmize these heuristics. In the case of conflicting heuristics, precedence is required on a case-by-case basis, preferably with multiple stakeholders involved.
Zuk & Carpendale (2006)'s heuristics (borrows heavily from Ware, Tufte, Bertin):
*ensure visual variable has sufficient length
*don't expect a reading order from colour
*colour perception varies with size of coloured item
*local contrast affects colour and grey perception
*consider colour blindness
*preattentive benefits increase with field of view
*quantitative assessment requires position or size variation
*preserve data to graphic dimensionality
*put the most data in the least amount of space
*remove the extraneous ink
*consider Gestalt laws
*provide multiple levels of detail
*integrate text wherever relevant
!!!!Comments & Questions
*A single case study application was evaluated using the three sets of heuristics. The tool in question supports certain tasks and is used for a certain type of situated analysis. It would have been nice to have compared the heuristics on more than one system, from another domain.
*Users were not domain experts for the tool used, but authors of the paper (expert reviewers from a usability, ~InfoVis tool development standpoint). (They advocate the participation of domain experts in future evaluations).
!![Faisal2008] - Grounded Eval. WIP
<<cite Faisal2008>>'s BELIV '08 workshop paper on [[Grounded Theory|Grounded Evaluation]] in information visualization. They conduct two studies, one a "usability" study (though not really, more of a benchmark speed and error study - testing the participants as much as the system, measuring effectiveness, efficiency, and user satisfaction). They provide a good summary of how to conduct a [[Grounded Theory|Grounded Evaluation]] approach evaluation. Their analysis of the study results, at the time of publication, was an ongoing work-in-progress.
Their coding (thus far) revealed high-level themes of usability (matching mental models, physical demand, insight (pattern identification, sensemaking), user background / technical experience, and user satisfaction / personalization / concreteness of the results.
!!!!Comments & Questions
*Appeared with <<cite Isenberg2008>>, which tackled the same subject. The authors claim the work is novel as of 2008 (no GT in ~InfoVis eval. prior? "Although we know of no cases where GT has been applied to the analysis of visualization user experience, it has been applied to a case study of the design of evaluation systems (Craft & Cairns (2006)".) This one offers a better summary of [[Grounded Theory|Grounded Evaluation]], but their comparative usability study is lacking rigour, and doesn't contribute much added value. ]
*What was the goal of their [[Grounded Theory|Grounded Evaluation]] evaluation? What did they seek to learn? Was it "How do they internalize what they see in the visualization?"
*The usability study didn't seem necessary, nor was it appropriate to compare to the [[Grounded Theory|Grounded Evaluation]] approach, as these studies have very different goals and measures.
**Their usability study is a necessary step in good design, but not novel research. Besides this point, why not [[Heuristic Evaluation]] or [[Expert Review]]? The tasks chosen for their usability study seem arbitrarily chosen - are these realistic of the domain?
**What ''research'' fields did the subjects come from? (this was not specified)
**How can they measure system efficiency without a benchmark? another system or method?
*Authors claim that [[Grounded Theory|Grounded Evaluation]] can allow researchers to tap into the //internalization// process of users using visualization tools, for accessing their subjective mental models, even those they can't describe verbally or are consciously aware of. I thought [[Grounded Theory|Grounded Evaluation]] was better suited for studying objective behaviour of individual and group work practices.
*Authors love acronyms.
*Vis tool figure needs better caption.
*What is the ''marking tool'' mentioned by the authors in section 5 - not clear. Could have used a figure. In fact the whole system could have used more figures - it wasn't clear what was possible with the system.
*Results of their ongoing GT analysis were so far vague and high-level, particularly the section on insight, touching upon sensemaking but digressing to discussion of dataset completeness and back to usability. Their theme of user satisfaction also covered many topics: personalization / concreteness of the results, none of which in great detail, overlapping with previous themes.
!![Tory2008] - Grounded Eval. Case Study (Architects)
<<cite Tory2008>>'s BELIV '08 paper reports on the field study of a building design group's planning meetings, a formative study without an existing visualization tool. They attended 10 meetings with this group as they collaboratively manipulated and discussed artifacts used in the building desing process (diagrams, charts, documents, sketches, etc.). Meetings were also videotaped. The conducted two forms of analysis on the data they collected. First, they quantitatively coded interactions using the video. Then, they conducted a [[Grounded Theory|Grounded Evaluation]] analysis of their observations and video recordings. The latter resulted in a hierarchy of thematic tags describing interactions. This led the authors to a much richer understanding of the interactions and the needs/requirements of the domain, relative to the quantitative video-coded results, as they questioned the validity of the numbers. The authors conclude by stating that qualitative analysis offers a holistic understanding of visualization use, and it's particularly useful for studying open-ended tasks where performance cannot be easily measured quantitatively.
!!!!Comments & Questions
*A better comparison of qualitative vs. quantitative methods than <<cite Faisal2008>>, who compares a quantitative usability study against a [[Grounded Theory|Grounded Evaluation]] observation study (different study goals and measures - one is "is the system usable?" and the other is "how do users internalize what they see in the visualization?"). The current work compares two studies with roughly the same goal: what interactions occur in this space? What is a typical meeting process? What do users need to do their job?
*Hating on reductionist scientific thinking: "Research questions and hypotheses cannot be precisely articulated at the beginning of the study". Also: "It is important to distinguish between qualitative analysis that is ad-hoc and qualitative analysis that follows structured methods of inquiry." To what extent can it be ad hoc? Some guidance is needed here. They later suggest that the former is formative while the latter is summative.
!![Tory2005] - Heuristic Eval.
<<cite Tory2005>>'s short article discusses [[Heuristic Evaluation]] with expert usability and visualization evaluators for qualitatively reviewing an ~InfoVis tool (regardless of design stage - summative/formative). They should compliment, not replace, other user studies and evaluation techniques. Heuristics should include those referring to usability and GUI design, and also those referring to visualization design and visual analysis tasks.
!!!!Comments & Questions
*They do not provide the heuristics themselves for usability (they cite the original Neilson paper), and they do not cite a set of ~InfoVis heuristics, (while there are a few - <<cite Zuk2006>> covers this).
*A good step-by-step guide to conducting a [[Heuristic Evaluation]] is provided.
!![Lloyd2011] - Longitudinal ~GeoVis Eval.
<<cite Lloyd2011>> conducted a long-term case study with 3 domain specialists (crime and disorder reduction) over the course of 3 years. Over this time the authors took a human-centred approach to desigining a geovisualization. Mixed (qualitative and quantitative) evaluation and design methods were used, some of which were more effective than others. They report their findings and offer advice regarding best practices for geovisualization design, which largely generalize to other application domains and visualization types.
They advocate studying real domain experts in context. They suggest a master-apprentice relationship with occasionally swapped roles, rahter than a consultant/client relationship: they should be viewed as co-discoverers, colleagues, and partners, rather than subjects. The authors suggest using real data, understanding its context of use.
''Method'': They began with methods for understanding context of use: [[Contextual Inquiry]] with seveal content analysis techniques (word frequencies, keywords-in-context, networks of relationships, card sorting (cluster analysis)). This was supplemented with interviews and observation, along with the study of internal and external domain documents. This was followed by a series of methods for establishing requirements: the [[Volere Method]] (a template of structure questions), a lecture on possible/suggested geovis and ~InfoVis design techniques followed by card sorting, user sketching, and delayed recall of concepts. This stage also included scenario interviews using real domain data, and a questionnaire. Qualitative and quantitative data was derived from each of these methods. At this stage, the sketching and scenario sessions were more effective than the template questions, the questionnaire, or the card sorting. The next stage involved early prototypes (paper), where an [[Autoethnography]] approach was taken to the design process, a highly introspective and reflective method. Recorded interactive sessions with these prototypes involved the [[Think Aloud Protocol]]. The use of real domain data is critical at this stage for cognitive plausibility. Production values should be kept low, encouraging the idea that the design prototypes are transient - sketchiness should be afforded. They suggest hybrid means for generating low-fidelity prototypes ([[Patchwork Prototyping]]), using mixes of paper and ~InfoVis toolkits such as [[Processing]] or [[ProtoVis]]. Later paper and interactive prototypes were more interactive and retained the use of real domain data. They were evaluated in sessions mixing [[Chauffeured Prototyping]] and [[Wizard-of-Oz Prototyping]]. Again the [[Think Aloud Protocol]] was used. 7 sessions with 3 experts lasted 2 hours each, followed by an interview. They found that paper prototypes generated more suggestions than interactive prototypes (except for interface-related improvements).
Methodologically, this long 3-year process placed a strain on the relationship with domain experts. What was learned from these sessions was highly data-dependent. An emphasis on iteration was very important. Free exploration generated more suggestions than task-based directed interaction. Fig. 7 outlines which techniques worked best and which techniques should be avoided. The list on p. 2506 outlines their key recommendations:
#design process should be interactive, creative, interesting, involvement of all stakeholders
#use a range of real data known to the domain experts, use it early in the design process
#emphasize transience in the designs generated (use paper)
#scenarios with data are effective for design suggestions, and also for ~InfoVis education
#develop digital sketches that are flexible and re-usable, used in conjunction with paper and interactive prototypes
#free exploration with prototypes is encouraged
#use the [[Think Aloud Protocol]]
#assume an attitude of co-discovery - do not establish the consultant-client relationship
#use an [[Autoethnographic|Autoethnography]] approach to design
#iterate within and between levels of the HC process
!!!!Comments & Questions
*I hadn't previously been aware of the term [[Autoethnography]] - seems like a good practice for any form of design work
*An exhaustive account of 3 years of work - many methods attempted and a thorough trace of each methods back to their original sources
*Paper prototypes generated more suggestions than interactive prototypes (except for interface-related improvements) - what were the other suggestions relating to? Visual encoding? Visualization type? Task flow? (they later state, albeit vaguely, that paper was good for suggesting functionality and enhancements).
*They continually compare their methodology against that of the Human-Centered best practice outlined in ICO 13407 - which pertains to all interactive systems. By comparing against the HC approach, which is so general, they do not address the requirements specific to visualization systems: providing insight, enabling open-ended exploration and learning, supporting decision-making and problem-solving under uncertainty.
*MS recommended this paper from ~VisWeek - to appear in Vol. 17, Issue 12 of Transactions on Visualization and computer graphics
*Related work was sprinkled throughout, as opposed to being presented in a dedicated section. The organization of the paper follows their design stages.
*Some parts and figures not especially clear / concise.
*In the concluding section (and Fig. 8), they argue that some low-fidelity design (cheap, paper prototypes with real domain data) is required before initial grounded context-of-use study (as suggested by <<cite Isenberg2008>>). Researchers should alternatively begin with design and proceed to requirements or context-of-use phases, before iterating on design and proceeding to evaluation. There is not much discussion of iteration and lopping through this cycle in the paper - they admit that their design process was fairly linear, much like the sections of the paper.
!![Winckler2004] - ~Bottom-Up Task Taxonomy
<<cite Winckler2004>> generated a hierarchical taxonomy that encompassed user goals, abstract tasks, interaction tasks, and application tasks (rendering tasks) supported by the system. Based on this taxonomy, they are able to generate scenarios for completing user goals by enumerating the paths of lower-level tasks, traversing the branches of the taxonomy. Once these scenarios are enumerated, they can be used as heuristics to evaluate the usability of a system, or to compare functionality and usability between several systems. The authors argue that scenarios generated from their taxonomy used for evaluation are more appropriate than ad-hoc or informal evaluation using ungrounded scenarios or ill-defined heuristics, which may be biased. They believe their method of creating a taxonomy (identify user goals, interaction mechanisms available, & rendering functions supported, then relate all three) will account for all possible user goal scenarios, and that these scenarios will be rationalized.
Abstract tasks, adapted from earlier taxonomy by Werhend and Lewis, include: locate, identify, distinguish, categorize, cluster, distribution, rank, compare, compare within and between relations, associate, correlate.
Their case study involves comparing 4 scenarios of a "locate system file" goal between a hyperbolic tree browser and a treemap browser.
!!!!Comments & Questions
*Step 1: producing unambiguous user goals and logical decomposition of subtasks seems trivial in the paper, the difficulty of this in reality is not addressed.
**What about scenarios in which the user has no explicit goal other than to explore a dataset, or the goal is ill-defined, such as "attempting to gain insight"
*Checking all possible scenarios of a task can run upwards of thousands of scenarios - the authors case study only checks 4 scenarios. How do they advise one should proceed given a goal with several thousand possible scenarios for completing the goal?
*distinctions between tasks, particularly abstract and user, and between application and interaction, or user and interaction, aren't as clear-cut as I expect the authors imagine. A user may engage in an abstract user task without interaction (adjusting a mental model, perhaps)
*The authors do not address the scalability of this method for large, complex systems
*paper/typos:
**CTT acronym not defined until p. 3 (not helpful)
**Table/Figure numbering off.
!![Springmeyer1992] - Characterizing Scientific Discovery
<<cite Springmeyer1992>>'s taxonomy of the scientific data analysis process, though grounded in the domain of scientific visualization, can apply equally as well to information visualization. It is superior to <<cite Winckler2004>>'s taxonomy in that it doesn't attempt to define tasks at the interface or sub-task level. It goes beyond interacting with visualization tools and discusses all aspects and phases of the scientific data analysis process at the abstract level, while bearing in mind opportunities for interactive systems to fill in gaps at these other phases. It recognizes that visualization is merely a means, not an ends to the process, and usually precedes hypothesis verification and further mathematical inquiry. Images are a by-product
They conducted several interviews and observation sessions ([[Interaction Analysis]], [[Contextual Inquiry]]), with 10 scientists working in a 7 domains: physics, biochemistry, numerical analysis. They aimed to presuppose as little as possible, and observe regardless an activity was supported by computer tools. Categories emerged from these sessions (observations being richer than interviews), becoming chronological lists, and structure arose our of these categories:
''Scientific data analysis'': distilling large amounts of measured/calculated data into simple rules/parameters, characterizing phenomena under study
*Investigation
**Interacting with representations: generating, orientation (cycling, translating, transforming), queries (high/low specificity), comparison, classification: further control for data exploration, particularly for quantitative information
**Applying math: calculations (estimations, slope approximations, transformations), deriving new conditions, generation of statistics (i.e. ANOVA)
**Maneuvering (organizing the data): navigation (mechanics of getting data into a system), data management / culling: can be distracting
*Integration of Insight
**Maneuvering (see above)
**Expression of ideas: recording, describing, (decision making): make judgements, conjectures, deductions: communication
Finally, the authors have several design recommendations:
*how can tools support scientists in integrating insight into their base of knowledge and experience?
*integrate calculations directly
*facilitate active exploration (qual. and quant.)
*capture the context of analysis
*link materials from different stages of a study
*minimize unnecessary or distracting navigation requirements
*support for culling large data sets: flexible, not distracting
*need for domain-specific knowledge
!!!!Comments & Questions
*Superior to <<cite Winckler2004>>'s taxonomy: doesn't adhere to a rigid grammar of tasks and subtasks, user tasks and application tasks.
*future work: collaborative scientific data analysis, very large simulation data
*scientific data analysis process is often not linear, so a chronological ordering of these tasks is doubtful
*methodology sounds awfully close to [[Grounded Evaluation]]
!![Kang2011] - GT Characterizing Intelligence Analysis
<<cite Kang2011>> conducted a longitudinal 10-week qualitative study with 3 teams of intelligence analysis students, following their progress on a strategic intelligence course project. Their instructor remarked that processes used were not dissimilar from those used by actual intelligence analysts working in the field. The researchers examined artefacts produced by the team, and conducted interviews and focus groups. They followed a [[Grounded Evaluation]] protocol and identified parallel, rather than sequential tasks throughout the course of the project: constructing and refining a conceptual model, data collection, analysis, and production. They noticed that some teams preferred a top-down model-driven approach while others preferred a bottom-up data-driven approach. The conceptual model need not be explicit or externalized.
Among their findings they emphasize the need for the support of collaboration and sharing, and that the intelligence process is not linear or sequential as <<cite Pirolli2005>>'s model would imply.
Their design implications include the following:
*externalize the thinking process
*support source management - pulled and pushed sources
*accommodate changing information sources
*support insight provenance and sanity checks prior to production of artefacts
*support asynchronous collaboration
*unify tools (all-in-one-solution)
!!!!Comments & Questions
*see ACH: analysis of competing hypotheses
*they still number their phases of the intelligence analysis process - an artefact of prior thinking?
*similar findings to that of <<cite Isenberg2008a>>, an observational study - who pointed out the irregular jumps (and inconsistent stage durations) across <<cite Pirolli2005>>'s model (no new findings here) -
*<<cite Pirolli2005>>'s model already accounts for many links and loops - however the authors stress that this model is focused on the information stages rather than on the task stages associated with intelligence analysis
*findings regarding collaboration appear obvious; the misconceptions are not often cited - who has these beliefs?
*externalizing the thinking process is not a tangible design implication
*what is insight provenance? the same as analytic provenance?
*the rationale for the all-in-one tool seems lacking - where is the evidence for this other than a couple of quotes (instigated by directed questioning)
!![Rodgers2011] - Eval. of Casual Vis for Energy Consumption
<<cite Rodgers2011>> report on exploratory research involving ambient (or peripheral) visualization of energy consumption in the home. This is in contrast with most exploratory design studies, which tend to involve visualization systems intended for attention-demanding centrally-focused visual analysis, exact data retrieval and comparison, and workplace contexts and workflows. Instead, ambient systems for the home must remain peripheral, aesthetically pleasing, and ecologically suited for a home setting, Nevertheless, the authors' design requirements dictate the need for usefulness and pragmatism, that the system still conveys accurate information on which the user is expected to make a decision (i.e. use less power).
There are several contributions made in this paper. Evaluation of these systems is inherently difficult given their requirements (peripheral, pleasing, yet informative). Their evaluation methodology involves several data collection methods and may lend well to other applications of ambient or casual visualization, particularly those designed to be aesthetically pleasing and contextually appropriate. Their designs themselves are also novel, making use of several visual encoding channels. The motivation and resulting requirements are also novel: other analytic systems can easily satisfy the requirement of informing users about energy use, however the use case involves focused, deliberate attention.
!!!!Comments & Questions
*~InfoVis group discussion 12.01.05
''Aesthetics''
*Aesthetics for the purpose of sparking curiosity or stimulating an emotional reaction. It would be interesting to determine the role of aesthetics in central-focus dedicated visual analysis tools to determine if this statement is true.
*Individual differences and idiosyncrasies: No matter how much design work goes into ambient, casual visualization / informative art - will users still never agree on aesthetic choices (i.e. colour schemes, etc.). Could a certain amount of pre-design critique and revision prevent this?
*Use of natural non-representational forms guiding the designs. Could abstract, non-natural forms worked just as well? It would be interesting to conduct a head-to-head evaluation. Note that the pinwheel design is not described as a natural form (but it is representational).
''Evaluation''
*How to evaluate designs that are not about productivity, but still result in decision-making and behaviour change (rather than purely informative art / lived with, rather than used)
*casual usage patterns (not limited to ambient visualization?)
*mixed methods user study in simulated domestic environments: do users really care unless they have a sense of ownership (an electrical bill to pay)?
*Obtrusiveness of experimenter / pygmalion effect - what happens when the experimenter leaves the room? Can a system detect when it is being looked at from across the room? How good / distant is gaze-tracking? (subjects looked at the display more often than they thought they did - would this still be true without the experimenter?)
*What other approaches could be taken in a longitudinal study to assess how the system satisfies the requirements? When novelty / curiosity wears off?
''Visual Encoding''
*Exploratory rather than confirmatory
*Their designs are based on the sole hypothesis that abstract, non-representational approaches are good candidates for ambient feedback. The selection of visual channels (motion, colour, placement, size, shape), are chosen according to design patterns for ambient visualizations (Pousman ref) - but which dimension gets mapped to which visual channel - is this ad hoc?
*re: pragmatic requirements - what was the criteria for understanding the visual mapping (correct / good / fair / poor / incorrect)?
*Use of motion - the transition between calming and stressful? (i.e. progress bars)
''Motivation & Requirements''
*Communicating a concern vs. showing data (above all else, show the data?); optimizing for visual efficiency vs. visibility and "interestingness", involving ambiguity in design, allowance of personal meaning. Despite this, participants claim the ambient feedback to be intuitive and saw the abstraction as a feature (but at a certain point, ambiguity undermines comprehension - what is that point with these designs?)
*Designing for behaviour change (unaddressed requirement?) Will the system help make decisions, engender behaviour change in the long-term. "Feedback operates on an unconscious level" - how do you assess whether or not this is happening?
*Interactivity - users wanted details on demand; customizability. What else?
*Could the display have the opposite effect? More energy use = more aesthetically pleasing visual entities? Positive feelings vs. guilt as a design requirement?
''Other''
*Ambient displays in workplace environments - group / individual presence awareness, project / job status, etc.
*How much information can be displayed in an ambient display? What is the limit in terms of visual encoding channels?
!![Sprague2012, Sprague2009] - Eval. of Casual Vis.
<<cite Sprague2012>> and <<cite Sprague2009>> relate to the evaluation of casual information visualization. The latter is a short methods paper, proposing techniques for evaluating the usability and utility of such visualizations in situ, with several case studies of casual visualization systems. The former, <<cite Sprague2012>> is a longer journal paper (read in ~InfoVis group 12/04/13). It uses a grounded theory methodology and a cultural probe data collection method to collect and synthesize user behaviour relating to casual visualization artifacts, including info-graphics, floor plans, advertisements, entertainment. Their analysis developed toward a model, which they call the [[Promoter-Inhibitor Motivation Model]] of visualization. This model accounts for how long a visualization is used and whether use is repeated, classifying uses as recognition (acknowledged but not used), single-use short-duration, single-use long-duration, repeat-use short-duration, and repeat-use long-duration. Where a visualization sits among these usage instances depends on promoters (motivating forces) and inhibitors (other forces), as well as user goals.
''Promotors'': personal interest, usefulness, curiosity, data correctness / trust, cost of misinterpretation, aesthetics.
''Inhibitors'': time constraints and higher priority tasks, artifact learning effort, insufficient data context.
Participant goals can be divided into ''extrinsic'' (social, avoid boredom) and ''intrinsic'', the latter being subdivided into ''learning and understanding'' (curiosity, information acquisition), ''utility'' (instruction, scheduling, task completion, orientation), and ''entertainment'' (humour, self-expression).
Based on this model, the paper provides design considerations for guiding use stages, recognition, designing for long-duration vs. short duration use, promoting repeat use. Personal relevance is critical, design for productive relaxation, reduce costs to initial use, data context is key, appeal changes over time, design for goal type, constrain user goals.
The authors conclude with a survey of related models for predicting use of technology, some being more applicable to casual vis than others.
Evaluating the model, broadening the model beyond casual vis, using the model to predict length and duration or use, and a survey of visualization systems using the model are left as future work.
!!!!Comments & Questions
*~InfoVis group discussion 12.04.13
*Is //casual// a domain? does domain matter? what about //setting//?
*Still not buying the rationale for separating casual vis from ~InfoVis argument, that due to the fact that users aren't getting paid (extrinsic goal), their motivations for using visualizations are altogether different. Professionally meaningful should also be personally meaningful
**I would argue that self-reflection is important in professional contexts just as it is in personal contexts
*By extension, would a GT study of professional vis users also turn up this model, or would it reconstruct Shneiderman's mantra, traditional vis wisdom?
*GT - was data collection and analysis interleaved? I don't think so.
**How / when did follow/-up validation interviews occur?
*Their data collection method (photos and diaries), still relied on recall to some extent - how to overcome this for a cultural probe study?
*Figures confusing, might have been an easier paper to read without figures
!![Ziemkiewicz2012] - Within/Between Graph Workflows / Individual Differences
<<cite Ziemkiewicz2012>>'s CHI note describes a short observational study of 4 immunobiologists using a scatterplot-based visualization tool called GenePattern. Their recorded observations of users' interactions were coded with a combined scheme of <<cite Springmeyer1992>>'s scientific data analysis and <<cite Amar2005>>'s low level visual analytical tasks. A second coding past was quantitative, counting instances and durations of visualizations used. Their findings revealed two markedly different interaction strategies, despite similar user goals and context. A //within-graphs// interaction strategy focused on individual data points rather than layouts, emphasizing information in context and visual connections between parts of the data, manual linked brushing and highlighting, with visualization as the primary output of analysis. A //between-graphs// interaction strategy was one in which switching between graphs was frequent, with each graph treated as a step in the analysis, constantly refreshing the view to see the data from different perspectives. The different strategies may indicate subtle differences in user goals, not evident at the start of the observation session: using visualization to increase confidence in results, to ensure the validity of the data, and efficiency (viewing lots of data).
!!!!Comments & Questions
*+qualitative coding strategies for observation sessions from <<cite Springmeyer1992>> and <<cite Amar2005>> a good use of prior theory; but could this be constricting?
*-small N (4); users evidently had different goals that were not communicated before the session began, despite being at a similar stage of analysis / workflow, and working with similar data.
*visualizations should record users' analytical processes, an analytical provenance, a history of past views or manual bookmarks such as in <<cite Heer2007a>>'s sense.us system.
*-analysis strategies may change within a session, within a workflow, reflecting local or global changes in user goals
*-cites <<cite Wickham2010>>'s graphical inference method, yet in the context of sequential analysis of valid data from different perspectives, not a parallel analysis of a valid view amongst random views. Not sure what was intended by this citation, perhaps a misreading/over-generalization of the graphical inference method.
*cites <<cite Pirolli2005>>, aims to bridge the sensemaking process to the immunobiology domain.
*Unsure where the CHI-level contribution is in this work - reads like a WIP / pilot study, not a lessons-learned paper
!![Willett2012] - Crowdsourcing social data analysis
#feature-oriented prompts: peaks / valleys / slopes / increases and decreases / micro-macro trends
#good examples: show off your expectations (increase in quality responses, but not significant)
#reference gathering - require valid corroborating evidence for analytical explanation (i.e. a news story) - leads to higher completion times, attrition, may not be necessary and does not guarantee higher-quality explanations; add a bonus monetary incentive for good references - EDA by definition is about speculation and hypothesis generation - reference gathering may be too restrictive
#chart reading (safeguarding against spammers and skimmers / people who don't care/understand the task) - easy questions about the data (a gold standard is needed, calibration of questions may be a lot of work a priori)
#annotation subtasks - callout regions / hotspots in charts: support ''deixis'': //refers to the phenomenon wherein understanding the meaning of certain words and phrases in an utterance requires contextual information// - wikipedia, Greek "to show/ to refer" (indexical usage), a weak interaction effect between prompts and annotations
#pre-annotating charts - focus on relevant details, but only if it is expected that crowd-workers will miss subtle detail in charts - may be limiting/priming/distracting, limits true EDA and possibility of novel insights, should only be used when obvious features are not important to the analyst, prompting viewers to look for more subtle cues.
#generate explanations iteratively by having crowd users review and rate others' explanations - alternating between annotating and rating subs task, good for detecting redundancy, as well as relevant, plausible, and clear explanations
<<cite Willett2012>>'s strategies showed improvement for domestic US workers, huge improvement for non-US workers;
!!!!Comments & Questions
*good presentation / talk at CHI; well-written/structured paper with good sections headings, sub-headings, problem-solution structure
*6 experiments! 1 of which a full factorial experiment with 16 conditions! (large experiments possible - participation rates described in <<cite Heer2010>>)
*explanation quality rating: how was this decided?
*what are novel insights? do speculative initial hypotheses count?
*focus on time-series (trends, peaks, valleys), but a few references to scatterplots (correlations and outliers), and bar charts (outlier detection)
**analog features for other chart types?
**what about cluster detection? model fitting?
**network analysis? clique detection, finding bridges, key nodes, central players, graph features (<<cite Kairam2012>>'s ~GraphPrism)
**parallel coordinates
**Overview - reference gathering news stories, tagging documents;deploy on a venue like ~ReportersLab rather than ~MTurk;
***crowdsourced tag rating
**novel visualization (with training, examples) such as <<cite Wood2011>>'s ~BallotMaps
*deployment in large company intranets, ask MS: <<cite Sedlmair2011>>
*how much a priori data mining to identify features in datasets/charts is necessary?
*strategies require some preliminary data analysis - how much time to invest? particularly pre-annotation, chart reading questions; administrators would require high-level review of complete dataset at the very least;
*how to deal with ambiguous data (if relevant features of a chart are ambiguous)
*Could be used in conjunction with crowd sourced - graphical inference (<<cite Buja2009>>) tasks: could these strategies help to weed out fake datasets from real ones?
**scatter plots, cluster detection in EDA, model diagnostics
!![Pohl2010] - Log file analysis of EDA
<<cite Pohl2010>>'s BELIV workshop paper presents findings from a study in which they evaluated use of a visualization tool intended for exploratory data analysis by means of qualitative and quantitative log file analysis. They discuss the role and characterization of EDA from the perspective of perceptual psychology and Gestalt psychologists: "//insight into the structure of a problem and by productive restructuring of the problem//".
Their study involved 32 computer science students performing 2 exploratory data analysis tasks designed by domain collaborators. Insight characterization was also performed alongside collaborators. Logs recorded their interface-level interactions with the tool. They performed qualitative and quantitative analysis of the log file. They found task-specific usage patterns, trends of macro-tasks (common sequences of interactions). They characterize some of these patterns according to Gestalt definitions of problem solving.
!!!!Comments & Questions
*summary: "we expected EDA to be random behaviour, but it wasn't! (mostly)"
*non-native English writers, hard to read at times; visualization tools referred to as methodologies or methods interchangeably.
*They do not address the insights, whether or not they were achieved by users, whether collaborators commented on their utility, or how different strategies allowed users to arrive at insight
*Student (non-domain) participants, fixed time limit on study, prescribed open-ended tasks, but prescribed nevertheless
*is EDA problem solving? an unstated assumption
*No full task-taxonomy is created, however they make references to several pre-existing task taxonomies without fully committing to them.
*Their hypotheses (䩠are based on what? Odd mixing of perspectives, a post-positivist perspective not really appropriate.
*"//subjects prefer to interact with the data and do not experiment with the visualization options//" - and "//users choose their preferred method of visualization early on and do not experiment with these options very much//" what, beyond the interface in question, are we to take out of these statements?
*interaction patterns may not result in insight - what to they result in?
*why did they believe that "//subjects would follow no particular pattern and use visualization methods randomly//"?
*one-off interactions not fitting in larger macro-task patterns were "//considered to be random user activities and were ignored//"
*quantitative analysis superfluous and didn't tell us anything that qualitative analysis didn't already; what are they referring to as "significant"?
*figure 4 is a mystery
!![<<cite Kandel2012>>] - Enterprise data analysis
Authors interviewed 35 enterprise analysts from a variety of domains (contrasted with intelligence analysts); methods were to characterize range, not to quantify prevalence of user types / challenges; authors adopted open-ended face-to-face interviews and an iterative coding method. User fall into categories of hackers, scripters, and application users (hackers and scripters have subtle distinction?) workplace contexts were also categorized); tasks framed as challenges:
*''Discover'': data acquisition, locating data, defining database fields
*''Wrangle'': format, parse from source files, integrate multiple formats / ingesting semi-structured data, aggregation and filtering
*''Profile'': check for suitability/ data quality, make assumptions, outliers, errors. distributions
*''Model'': summarize, predict, summary stats, running regression models, performing clustering/classification, feature selection, sampling/scaling
*''Report'': communicate insights and assumptions to consumers
Other issues: data migration, operationalizing and saving/sharing workflows.
Future trends include availability of public data, the rise of hadoop, the demand for hacker analysts, and a need for collaborative analysis. Design implications include workflow breakdowns/saving/sharing and capturing metadata, scalable analysis tools/techniques, helping out those with little programming experience.
!!!!Comments & Questions
*Not visualization-centric, as tasks involve data acquisition, wrangling, integration; many analysts don't rely on vis
*results unsurprising
*tasks not taxonomic, but sequential and representative; each contain examples
*Nice interview methodology; would like to see their supplementary materials
*well written, sections reporting user profiles and contexts, abstract high-level tasks (though framed as challenges), future trends (were these based on interviews or on being in-the-know?) and design implications (based on both interviews and future trends
*collaboration disconnect: supervisors want more collaboration and sharing (intermediary data, scripts) among subordinate analysts; analysts want to stay independent, overhead of sharing too costly. Will bringing down the overhead lead to sharing? Many analysts are do-it-yourself-ers;
*future trends section reads like job description for future big data analysts
*utility of fig 1? Aim was not to assess prevalence of user types / challenges
!![<<cite Erickson2013>>] - Domestic Energy Consumption Portal - Longitudinal Deployment & Eval
The authors (IBM TJ Watson) conducted a 20 week pilot project in which an electricity consumption tool, a web-based visualization, was used by 765 households in a small Iowa city (Dubuque). They collected usage data and electricity consumption data, as well as survey and interview data from a subset of these households (due to privacy constraints, these data were kept separate).
They set out to understand the utility of the tool and how it provided energy consumption feedback, incentives for changing behaviour, goal-setting challenges, and opportunities to chat and converse with other users. The tool allowed a limited amount of drill-down analysis with alternative views, facilitating multiple types of comparison.
They provided ample training and support for the tool (telephone and web-based Q&A), in-person training sessions and town hall meetings.
The response rate to the web-based survey was 22%. The usage rate of the system. Usage logs show that 266 off 765 households used the tool. 18 households were interviewed.
The timeline-based views and time-based metrics of the tool were deemed to be most important and were most used. Social features (chat, comparisons with neighbours) were the least-used features. Households didn't appear to be motivated by competition, but rather comparing against their own baseline. This was apparently not due to privacy concerns. Comparisons and point systems were viewed as unjust since participants weren't rewarded for prior good energy usage behaviour or energy saving measures already taken prior to beginning the study. Many users found logging in to the tool difficult or cumbersome.
Participants' energy usage behaviour did change in some households, most often shifting electricity use to non-peak periods, using energy for shorter durations (e.g. shorter showers).
!!!!Comments & Questions
*How complicated will these portals become when device-level data is possible with the advent of smart meters and networked devices?
*Do findings extend to non-residential commercial customers? Do they want to compare against their own baseline (as in residential users), or are they interested in comparing against neighbours? residential users didn't care to do the latter.
*Re: opt-in: Doubt that the the non-response bias would extend to commercial customers who are likely to be more interested in cutting energy expenses. User engagement problems may not extend beyond residential users.
*Nice linking of Fig. 2, 3 with annotated callouts.
*The tool doesn't take into consideration outdside temperature, nor can users normalize by outside temperature.
*The tool doesn'e explain peaks / anomalies in energy usage.
*What's the population of Dubuque?
*A decent and lengthy discussion of their methodology's limitations
*Given the lack of adoption of social features of the tool, it is not clear how to achieve social / neighbourhood energy reduction goals with a system like this.
!References
<<bibliography>>
!![Sutcliffe2000] - Lessons Learned re: Wrong Methodological Choices
<<cite Sutcliffe2000 bibliography:Bibliography>> evaluated a novel visualization for displaying search results in the information retrieval process. They conducted a largely quantitative study with mixed methods and multiple dependent variables collected with unrepresentative, unmotivated study participants on a set of predefined tasks. Unsurprisingly (in 2011),, but unexpectedly for the researchers, they had inconclusive and mixed results, where results of some methods (empirical task results, user preferences) didn't agree with results from another (interviews, questionnaires). They discuss the role individual differences played in their study and discuss their methodology.
''Methodology'': user search strategies were examined using a novel system of displaying search results (akin to a visual thesaurus). 12 users were recruited from the university community. After a background questionnaire and a training session with the tool, 2 Predefined information retrieval tasks were given. They were encouraged to use the [[Think Aloud Protocol]]. They completed the tasks and filled out a usability questionnaire, then they were interviewed about their use of the tool and their understanding of how the tool functioned.
''Results'': a mismatch between all the sources of data was apparent: empirical results (relevant information retrieved) were poor across the board (authors predict poor motivation and low domain knowledge). The system's usability was rated highly. Observation of the participants revealed usability problems as several critical errors were made. The interview revealed many more usability problems as it became clear that the system was poorly understood and didn't correspond with participants' mental models, explaining the sub-optimal performance (not in terms of errors, but in sub-optimal search strategies). They analyze several common behaviour patterns with their system and suggest an ideal pattern performed by one of the top-performing participants. They suggest that their participants were "cognitively lazy".
!!!!Comments & Questions
*After reading more recent literature (particularly in the BELIV space), this seems to be a warning paper about study design, study goals, and research questions.
*The tool was not evaluation-ready, as it didn't support multitasking or many of the users' mental models (or their training was insufficient)
*An unrepresentative set of users who were poorly motivated, only using the tool for a few minutes. These results should be expected. Their goals were not clear: was this just a usability test? (then its' not research). Was this novel visualization tool better than the state of the art? (no head to head comparisons). Were they trying to solve a domain-specific problem, find a type of person who would be responsive to the tool, a context appropriate for the tool? (wrong study to run, these issues were not addressed). This paper reminds me of the opinions in <<cite Ellis2006>> as to what constitutes an appropriate evaluation for the purposes of conducting research.
*I'm not sure what value the behavior patterns brought added to the paper, other than drive home the point that participants just didn't understand the system, except one or two of them who evidently think like the researchers/designers.
!![Chang2010] - ~Learning-Based Eval.
<<cite Chang2010>>'s BELIV '10 paper describes a summative quantitative evaluation technique that hinges upon learning and re-application of analysis and system-usage skills to a new problem domain. They also offer critique for their approach and indicate how it may complement other evaluation approaches, particularly qualitative approaches such as those described by <<cite Isenberg2008>>. They argue that their evaluation measures the practical use of an ~InfoVis tool.
They also offer critique for techniques for quantifying insight (<<cite Saraiya2004>>, <<cite Saraiya2006>>), which is inherently hard to define, as there is too much variability as to what constitutes a discovery, and for measuring productivity (Scholtz (2008)): the productivity metric accounts for how much information is viewed, not for how much is absorbed.
Their technique evaluates how well a user can solve a new task with a new data set once they have learned the visualization and spent time solving tasks with a training data set, a re-application of knowledge. Their technique does not quantify insights. After initial task assignment (designed by a domain expert), the user's open-ended exploration with a system is recorded, but doesn't constitute dependent measures for the evaluation, but recorded actions do however provide an explanation as to how a system is learned. It is the next stage, a testing phase, which assesses learning. Their approach is akin to an open-book and closed-book test. First, a task is given along with a new data set (open book). Then, a semantic questionnaire about the data set is given (closed books). For scoring, they propose using data sets from the VAST challenge, where ground truth is known for the data.
Their critique for their own technique centers around the problem that what is learned about the ~InfoVis tool, what is learned about the data set, and what is learned about the task (skill transfer), are not easily separable or easy to measure, thus confounding the results. Using known datasets (such as the VAST datasets) can help to isolate out what is learned about the data. Separating what is learned about tool use and what is learned about the task is more difficult and remains an open question.
They propose adoption of their approach in the evaluation of future VAST contest entries, in which each submission is evaluated at the conference with a new data set. They also describe how this technique could compliment [[Grounded Evaluation]] approaches (<<cite Isenberg2008>>), and how their approach could be applied during design and in the post-design phase.
!!!!Comments & Questions
*My favourite BELIV paper to date. However, measuring learning opens up the literature space even wider.
*This works well for well-defined tasks. What about open-ended exploratory tasks? Tasks that are ill-defined or non-directed? How do you assess skill transfer for these tasks? But first, who performs open-ended exploratory tasks?
*Open question: how do you isolate and measure what is learned about using an ~InfoVis tool vs. what is learned about an analysis task, one that is tool-independent?
!![Borkin2011] - Expert Eval., ~A-B Comparision
<<cite Borkin2011>> challenged the status quo of artery visualization: 3D, rainbow colour map. These are difficult to interact with and occlusion can be problematic. Existing 2D visualizations do not maintain anatomical structure and relative position (a rectangular projection of an artery cross-section). They conducted a formative qualitative study to determine the tasks that medical professionals (cardiology and radiology specialists and researchers) do when using these existing visualizations. The researchers began designing and getting feedback on 2D visualizations that maintained anatomical structure and relative position of arteries using a tree structure. They also experimented using several alternative colour maps (in addition to rainbow).
Users were unconvinced that a 2D visualization with non-rainbow colour map would do better than their existing methods, so the researchers conducted a formal [[Laboratory Experiment]] to determine the best method for locating regions of stress in the artery walls (accuracy, speed, and efficiency were dependent measures). Informal feedback (questionnaires and interviews) were also gathered. To no surprise, their solution (~HemoVis), a 2D tree visualization with anatomically correct layout and a red-to-black colour map (white at divergent point) was the clear winner. With this result, the authors were able to convince medical professionals that the status quo was slow and error prone, and that their new method should be adopted, contributing to better treatment of heart disease.
!!!!Comments & Questions
*//"rainbows kill people"//
*[[Stephen Few's blog post |http://www.perceptualedge.com/blog/?p=1090]] about this paper
*colour map was a between subjects factor, dimensionality was within, but analyzed separately with t-tests. The resulting data was not normally distributed, so non-parametric tests were used. Could it be that they weren't able to run a non-parametric mixed-factor ANOVA? Perhaps Wobbrock and Findlater's aligned rank transform method (ART) <<cite Wobbrock2011>> could have been used here.
!![Micallef2012] - crowdsourcing the effect of visualization for Bayesian reasoning - ~InfoVis '12
<<cite Micallef2012>>'s ~InfoVis / TVCG paper describes two studies in which the results indicate that augmenting textual Bayesian reasoning questions with various visualizations does not improve response accuracy or reduce bias. These results, accumulated via ~MTurk with a large sample size from a general population, are in contrast to previous lab studies, typically performed with small sample sizes and with student participants, who tend to be more familiar with Bayesian reasoning.
In a second study, Micallef et al. found that removing numerical elements from a textual question prompt reduced response bias, though accuracy was not improved. This suggests that visualizations are more effective when used actively as a primary source of information, and not passively or supplementally to a textual question prompt. The studies also suggest that subjects cannot reliably recall their own thinking process, tending to ascribe more of their response to the visualization than is justified.
The two studies also call into question the validity of ~MTurk studies as used in visualization and HCI research. ~MTUrk affords large sample sizes and short experiments involving low-level perceptual tasks, such as in <<cite Heer2010>>, where the tasks require little to no training and engage perceptual activity common to all sighted humans. Bayesian reasoning problems, at least when posed in a textual form, require training and a mathematical understanding, and thus operate at a higher level of cognition than the level of visual perception. So it is not surprising that a group of participants sampled from the "general" ~MTurk population fare poorly when approaching these problems. The study also indicates the power that textual instructions or prompts have over individuals, particularly when containing numbers. Individuals see these prompts as having more authority than supplemental images. And yet they are generally bad at making logical inferences, however believing that (a) they are better than they actually are and (b) the inclusion of supplemental images boosts their confidence.
What the authors accomplished in the 2nd experiment is that they substituted a perceptual inference for a more difficult logical one <<cite Casner1991>>; they showed that a perceptual inference has to be primary and forced, not secondary, supplemental, or optional.
!!!!Comments & Questions
*Demographics of Turkers in experiment 2? gender/age/education?
*No individual abilities reported for exp. 2?
*Claim of crowdsourcing population more diverse than university students?
*In experiment 2, subjects were less accurate overall, reasons unclear
*An important case of negative results that should be shared.
*Has broader implications for storytelling in visualization, embedding visualizations in text. People will use and believe numbers in the text rather than scrutinize the visualization. The visualization will be seen as corroborating with the text, it adds support and authority, but the numbers in the text could deliberately mislead and the visualization will not be challenged. A possible follow-on experiment could attempt to isolate and identify how textual information misleads people.
*They measured the numeracy and spatial abilities of participants; were these representative of the general population for both genders? Do such figures exist? Their findings were not supporting of high/low individual differences.
!!!!~VisWeek notes:
*Example of woman with positive mammography test result (P misdiagnosis 10%), what is the probability that she actually has cancer? (P Cancer in population 1%)
**many doctors get this wrong
*many representations are intended for specialized students, not area-proportionate
*crowdsourcing study to explore 6 alternatives to visual representations for this (163 workers)
**subjects confident but often wrong, no difference in error amounts b/w alternative visualizations, no visualization control group
**89% found diagram helpful, but doubted it?
*solution: change the text prompt (task), 2nd study (460 workers)
**most effective textual representation is to provide no numbers in the instruction, numbers throw us off; least amount of errors
*simply adding a visualization to a text doesn't help, but a diagram can be helpful if numbers are removed from texts
!!!!!Notes:
*~EulerGlyphs Bayesian reasoning vis
*claim that ~MTurk is more like the "real world"
*word cloud closing slide!
*explanation for removing numbers?
!!!!!Questions:
*''Caroline Ziemciewitz'': visual/numerical/spatial literacy effects?
**no sig. effects; deferred to some thesis
*''Q'': ~MTurk replication? Effect of culture (~MTurkers from India)᭢led) (out of time)
!References
<<bibliography>>
[>img(60%, )[Information Visualization Pipeline|http://lh4.ggpht.com/easy.lin/SGa2VM1RN1I/AAAAAAAAAXQ/E_ukOvol8ks/pipeline.png]]
!!!!Sources:
*Card, S.K. & Mackinlay, J. D. & Shneiderman, B. (1999). //Readins in information visualization: using vision to think//. Morgan Kaufmann Publishers Inc.
*[[Prefuse Visualization|http://prefuse.blogspot.com/2008/06/pipeline.html]]
!![Wood2011] ~BallotMaps
[>img(40%, )[BallotMaps|http://www.visualisingdata.com/blog/wp-content/uploads/2011/10/ballotmaps-591x450.png]]
<<cite Wood2011 bibliography:Bibliography>>'s 2011 ~InfoVis paper presents a design study aimed at uncovering name bias in alphabetically ordered ballot papers for a recent London, UK municipal election. 3 political parties were represented each by 3 candidates in the majority of the city's 614 wards, which are subsumed by 32 boroughs. The researchers sought to determine if the alphabetic ordering of candidate names on a ballot and the inferred ethnic origin of the candidates' names had a biasing effect in terms of the number of votes received and whether the candidate was elected. Geographic position of the boroughs and wards were also to be considered. They developed ~BallotMaps, a space-filling conditioning visualization, whose underlying grammar could be configured to display combinations of attributes, those being numerical (number of votes), binary (elected/not-elected), ordered (alphabetic position within a party), categorical (party alignment), geospatial (relative location of borough). Derived attributes were also considered, such as the signed chi score of whether the number of votes received was more or less than expected, and the residual score of whether the number of votes received was more or less than expected, not considering name order. The latter would be used to determine if a name bias existed as a result of factors other than the alphabetic placement of the name on the ballot, such as inferred ethnicity or geographic location. The visualization allowed them to answer their initial research questions, such as finding geographic locations where alphabetic name ordering had (or did not have) a biasing effect. They also found outliers, which prompted new queries using derived attributes to examine name and ethnicity effects.
!!!!Comments & Questions
*+flexible underlying grammar, ability to handle original and derived attributes to answer complex research questions, including interaction effects and geospatial dependencies
*+space-filling overview visualization
*+addresses public dataset problem: available 㥳sible 䥲standable
*+visualization for political transparency (cites Munzner position paper)
*+visualization has potential for drill-down interaction from boroughs to wards, overview and detail, side-by-side comparison of multiple wards/boroughs (user-selected and geospatial arrangements)
*+grammar has potential for integrating additional attributes: geospatial SES, census, ethnicity data
*+usage potential for large scale deployment in 2012 US federal election
*+use of <<cite Wickham2010>>'s graphical inference technique for finding other sources of name bias, using tag clouds: random samples of first-place alphabetical names vs. first-place names with negative residuals (lower than expected number of votes based on alphabetical placement)
**-note clear immediately how this was deployed: how many evaluators? how many iterations? Rorschach vs. lineup method? methodological details left out
*-geospatial arrangement not veridical, potential for misleading mismatch between relative and actual size of borough, number of votes per borough, number of wards in each borough; a possible compromise solution could allow toggling between space-filling normalized visualization (current), choropleth maps, and maps with glyphs
*-//geographical area [᳠little bearing on the voting behaviour// (p.3 3.1 bottom para) - claim known a priori - in direct opposition of what is later shown and discussed
*what was the project inception? No domain researchers. Based on a suspicion/hypothesis/hunch (but from where? media, public opinion, political bloggers, conspiracy theorists?) - was project inception reading the cognitive psych. literature on primacy/recency effects?
*was fig. 5 the initial project inception? no complex visualization needed!
*research questions: ~RQ1,2 (vis not necessary, only stats); only with ~RQ3, 4 does visualization become useful; ~RQ4 likely came afterwards, after initial data analysis and discovery of outliers and deviations from expected election result after the alphabetical name ordering factor was removed - did this RQ arise serendipitously? 3.1 p.6 - was there really no precedent for alternative sources of name-related biases? I find this hard to believe
*fig. 1: vertical size of parties not equal across boroughs - were candidates not running for election in all wards? not clear. Individual wards not apparent.
*ethnic name bias for English/Celtic names but not for international names the opposite of what I expected: was finding supported statistically?
*fig 7. hard to spot trends referred to in the discussion
*fig 8: cells within regions: is arrangement reflecting veridical geospatial arrangement of wards?
!![Pinaud2012] PORGY
[>img(40%, )[PORGY|http://tulip.labri.fr/TulipDrupal/sites/default/files/uploadedFiles/users/21/overview_annotated.png]]
<<cite Pinaud2012>>'s ~EuroVis paper presents a design study focused on graph rewriting, with 2 domains serving as case studies, working with the deployed tool. The paper frames the abstract domain models and tasks, application features (visual encodings and interactions), and algorithms within Munzner's nested model framework.
!!!!Comments & Questions
*+Table 1 columns match Munzner's nested model structure, however not immediately clear what the rows denote, really nice and succinct breakdown of abstract domain tasks, concrete application tasks and encodings, and algorithms
*Graph rewriting domain highly theoretical, difficult to understand without background; molecular biochemistry case study easier to grasp than theoretical functional computing case study
*Multiple views, linked interaction, analytical provenance and future branching possibilities within graph derivation tree
*Future development to include Focus+Context interaction of the graph derivation tree, supporting local and global graph inspection.
!References
<<bibliography>>
!![Steinberger2011] Context Preserving Visual Links
[>img(40%, )[Context Preserving Visual Links|http://www.perceptualedge.com/blog/wp-content/uploads/2011/10/visual-links.jpg]]
<<cite Steinberger2011 bibliography:Bibliography>> designed a technique for linking related focus areas across multiple windows and applications. Their technique uses curved lines overlaid over the displays, but without occluding other salient contextual information, thus preserving context. This is done by computing a visual saliency map of the display and adjusting the path of these connecting lines such that they are drawn over areas of low saliency, where presumably no important contextual information is displayed. Color is also adjusted dynamically to contrast with the display areas that the lines traverse. Where occlusion is unavoidable, the alpha value of the link lines are adjusted such that contextual information (i.e. text) is still readable. They extend the technique to the display of multiple connected sets of foci using bundles of visual links and discuss how the algorithm computes the link path.
''Evaluation'': they conducted a controlled, quantitative [[Laboratory Experiment]] to assess the usability applications using these visual links. They compared performance on a visual search task between Context Preserving Visual Links, naive direct visual links (where context occlusion occurs) and no visual links ([[Brushing]] using coloured frames around related foci). They also collected gaze data and questionnaire and interview responses. Both types of visual links outperformed no visual links. Gaze patterns were systematic with no visual links, and far more efficient with visual links. Performance was no worse with Context Preserving Visual Links than with naive visual links. Users generally liked Context Preserving Visual Links, despite it adding more clutter to the display.
Some examples of [[Context Preserving Visual Links|https://picasaweb.google.com/111174118588927995482/Visual_links?authkey=Gv1sRgCLLGwr2c1NDcag]].
!!!!Comments & Questions
*''Best paper award, ~VisWeek 2011''.
*Some discussion of this work: [[Stephen Few's blog post|http://www.perceptualedge.com/blog/?p=1090]], and [[response from the authors|http://alexlexvisual.blogspot.com/2011/10/context-preserving-visual-links.html]]
*They didn't compare performance on a visual search task with no [[Brushing]] at all, an absolute baseline. Furthermore, they only compare one style of [[Brushing]] - they didn't compare against situations in which contextual information is dimmed, blurred, or de-coloured. They discuss how these techniques are not well-accepted by users in their related work section.
!![Brandes2012] Gestaltlines
<<cite Brandes2012>> is a follow-up to last year's ~VisWeek paper: "Asymmetric Relations in Longitudinal Social Networks", a paper about these of inline glyphs / glyph matrices.
This (submitted) paper extends Tufte's sparklines to incorporate multivariate data / glyphs, read and perceived according to Gestalt principles. They present 3 uses of Gestalines using examples of data from the literature (both original and derived multivariate data): phase shifts in population dynamics, sports win/loss records, and building traffic in/out. Then they apply Gestaltlines to a novel multivariate dataset regarding gambling and risk behaviour. They iterate through several design considerations before arriving at a Gestaltline representation that answers some of their emerging questions about the dataset. Formal evaluations and graphical perception studies are to follow.
!!!!Comments & Questions
*Sparklines are annotated by surrounding text, which directs attention to salient features of interest; they are not intended for exploration or lengthy analysis.
*authors claim sparklines are predominantly univariate time-series data; their examples are categorical - what about multivariate time series stock tickers?
*are filled polygon glyphs their own invention or Chambers 1983?
*Ware's description of glyph perception superior; theirs is confusing
*Gestaltines' 1st example is spatial/categorical rather than multivariate time series
*Example 3.1 - as a reader, you are primed to look for emerging pattern (a derived attribute, relative difference in change), unfair comparison; gestalt line involves switching in reading direction from L - R to diagonal UL - LR; derived vs original attributes a separate issue?
*3.2 - less ink, rule more complex: graphical perception study required here - filtering is non-obvious; authors claim emphasis is on wins - isn't is on deviations from expectations?
*3.3 - new finding in geyser activity not apparent (even with colour)
*3.4 - rows of gestaltlines no longer appearing in line with text, without helpful annotation or directed attention: if filling up that much space, why use minimal gestaltlines? why not go for van Wijk / van Selow's calendar/cluster view to detect periodic disruptions?
*case study in section 4: good sample populations; design decisions hard to follow; hard to keep track of when attributes were combined / derived / filtered; I thought resulting fig.6 / final design would have colour nodes indicating wins/losses - why were these filtered?
**questions about dataset became more focused during design iterations - hard to disentangle these;
*I'd be curious to see results of graphical perception study on case study, geyser, and sports data.
!![Kairam2012] ~GraphPrism
[>img(40%, )[Context Preserving Visual Links|http://vis.stanford.edu/images/figures/graphprism.png]]
<<cite Kairam2012>>'s AVI paper presents the iterative design an application for simultaneous analysis of several properties of large network structure. They adopt Bagrow et al's [R3]'s //~B-Matrix// technique and present stacked histograms of network properties side-by-side with a node-link graph. They began with paper prototypes, adopting <<cite Wickham2010>>'s Graphical Inference "line-up" technique as part of the evaluation of their design, where network analysis domain experts acted as participants. They also include a prescribed task evaluation (a sorting/classification task). They proceed to develop an interactive prototype, which was used in a co-authorship network case study.
!!!!Comments & Questions
*+iterative design, multiple evaluation techniques and a case study
**-use of prescribed task in paper prototype evaluation
*+domain expert users in initial evaluation
*+usage of graphical inference "line-up" technique with methodological details (unlike <<cite Wood2011>>).
!![Wickham2010] A Layered Grammar of Graphics
<<cite Wickham2010b>> describes a grammar for creating static plots in the R programming language, beginning with basic plots, describing the basic components of the grammar and how it differs from language-agnostic grammars of earlier work; a layer may consist of:
*data and aesthetic mappings
*geometric objects (points/text (0D), paths/lines (1D), polygons/intervals (2D)
*position adjustment
Layers share:
*scales (continuous/categorical/interval/binary)
*facet specification (small multiples)
*statistical transformations
*coordinate system (Cartesian, semi-log, polar)
After introducing the grammar along with examples, longer examples and reconstructions of well-known plots serve as case studies (e.g. Napoleon's march on Moscow, Nightingale's Coxcomb plots).
Finally, a discussion on the //poetry of graphics// is discussed. If invalid plots are prohibited, as a spell-checker would warn for incorrectly spelled words, could there be a grammar checker? It could act as a safeguard against too many variables, overplotting, alphabetical ordering, polar coordinates (evil pie charts).
!!!!Comments & Questions
*Does not yet include interaction in the grammar: a foreseeable problem with linked brushing: involves //interaction with the underlying data, and incorporating this type of interaction into the grammar will require deeper thought//
!![Dunne2013] Motif Simplification in Network Visualization
<<cite Dunne2013>> (Proc. CHI '13)
!!!!Comments & Questions
*discussed in InfoVis group meeting, April '13
!![<<cite Aigner2011a>>]
<<cite Aigner2011a>>'s textbook on the visualization of time-oriented data:
!!!Time and ~Time-Oriented Data
Ch. 3's design aspects of time-oriented data:
*''time'': scale (ord / discrete / continuous), scope (point / interval), arrangement (linear / cyclic), viewpoint (ordered, branching, multiple perspectives); //abstractions//: granularity and calendars (none / single, multiple), time primitives (instant / internal / span), determinancy (determinate / indeterminate)
*''data'': scale (quant / qual), frame of reference (abstract / spatial), kind of data (events / states), number of variables (univariate / multivariate)
*''data and time'': internal time (non-temporal / temporal), external time (static / dynamic)
!!!Visualization aspects
Ch.4: - see task-related [[notes|Task Characterization: Meta Analysis]].
!!!Interaction Support
Ch. 5 on interaction in visualizations of time-oriented data, describing the intents of interaction using the definitions of <<cite Yi2007>>: //select, explore, reconfigure, encode, abstract/elaborate, filter, connect, undo/redo//. Also discusses <<cite Spence2007>>'s notions of //stepped// vs. //continuous// interaction, also discussed <<cite Norman1988>>'s stages of action, as well as the model-view-controller pattern
*''basic interaction'': overview + detail, focus + context, semenatic and graphical zooming, multiple views
**''direct manipulation''
**''brushing and linking''
**''dynamic queries'' - incl. graphical selection + sketch-based dynamic queries (see Hochheiser / Shneiderman '04, Holz and Feiner '09)
*''event-based visualization'': (see Tomniski '11) combines interactive, automatic, visual methods for generating targeted representations catered to a user's task (a directly specified query / search for a particular pattern); they are not appropriate for undirected queries / pattern detection - these representations must :
**communicate that something interesting has been found,
**emphasize interesting data among the rest of the data
**convey what makes the data interesting
!!!Analytical support
Ch.6: - see task-related [[notes|Task Characterization: Meta Analysis]].
!!!Survey
Ch. 7 is a survey of visualization techniques for time-oriented data, categorized along several variables:
*data
**frame of reference (abstract / spatial)
**variables (univariate / multivariate)
*time
**arrangement (linear / cyclic)
**time primitives (instant / interval)
*vis
**mapping (static / dynamic)
**dimensionality (2D / 2D)
The survey is also available as an interactive web application at [[timeviz.net|http://www.timeviz.net/]].
Some of the interesting and noteworthy examples (full citations [[here|References: To Read]]) include:
*~TrendDisplay (Brodbeck and Girardin - Proc. posters ~InfoVis '03): abstract, univariate, linear, instant, static, 2D
*~ArcDiagrams (Wattenberg - Proc. ~InfoVis '02): abstract, univariate, linear, instant, static, 2D
*Silhouette Graph, Circular Silhouette Graph (see Harris' //Information Graphics: A Comprehensive Illustrated Reference//, Oxford Univ. Press '99): abstract, univariate, linear / cyclic, instant, static, 2D
*Cycle Plot (Cleveland's //Visualizing Data//, Hobart Press '93): abstract, univariate, linear / cyclic, instant, static, 2D
*Tile Maps (Mintz, ~Fitz-Simons, and Wayland - Proc. SUGI '97): abstract, univariate, linear / cyclic, instant, static, 2D
*Multi Scale Temporal Behaviour (Shimabukuro, Flores, de Oliviera, Levkowitz - Proc. CMV '04): abstract, univariate, linear / cyclic, instant, static, 2D
*GROOVE (Lammarsch et al - Proc Intl. Conf IV '09): abstract, univariate, linear / cyclic, instant, static, 2D
*~TimeSearcher (Hochheiser and Shneiderman - IV '04; Buono, Plaisant, Khella, and Shneiderman - Proc. VDA '05): abstract, multivariate, linear, instant, static, 2D
*~LiveRAC (~McLachlan, Munzner - Proc. CHI '08): abstract, multivariate, linear, instant, static, 2D
*~CircleView (Keim, Shneidewind, and Sips - Proc. AVI '04): abstract, multivariate, linear, instant, static / dynamic, 2D
*~FacetZoom (Dachselt and Weiland - Proc. CHI '06): abstract, multivariate, linear, instant / interval, static, 2D
*~Time-Oriented Polygons on Maps (Shanbhag, Rheingans, and Desjardins - Proc. ~InfoVis '05): spatial, univariate, linear / cyclic, instant, static, 2D
!!!Conclusion
Ch. 8 concludes and mentions several ongoing application and research challenges: of deploying visualization tools into existing workflows and analysis contexts, of providing user support and guidance, of conveying data quality and provenance, of scalability, novel interaction methods, advanced time-oriented analytical methods and the furthering of the field of visual analytics, collaboration and multi-display environments, accessiblity, and evaluation.
!!!!Comments & Questions
*see interval and absence visual query specification language in Munroe/Shneiderman/Plaisant (Proc. CHI '13)
*event-based visualization vs. dynamic queries? is the former unique only in that it generates a novel representation, whereas dynamic queries do not change the representation?
!References
<<bibliography>>
An evaluation protocol defined by <<cite Saraiya2004 bibliography:Bibliography>>, where ''insight'' is defined as a unit of discovery, an individual observation about the data. The methodology was refined in subsequent work: <<cite Saraiya2006>>, <<cite North2006>>, <<cite Saraiya2010>>, <<cite North2011>>.
Researchers (including a domain expert) score insights, characterized by the fact discovered, the time to reach the insight, the domain value of the insight, whether the insight led to the formation of a new domain-relevant hypothesis, whether the insight pertained to breadth or depth, whether it was the result of directed inquiry or it was a serendipitous discovery, its correctness, and the domain-relevant category it fell under (categories are not predefined, they are determined after the data is collected).
Dependent measures include the tasks undertaken by participants, the total domain value of all insights and their domain-specific categories. You can also collect user engagement and motivation, as well as comments regarding usability and visual encoding.
The method can also be used in a longitudinal study. This has pros and cons. While you can use a real data set and recruit participants with greater domain expertise, you will lose the timing granularity of insight generation, the rate of insight discovery. You may also lose the ability to observe participants, gauge their levels of motivation and engagement. However, scoring insights gained longitudinally may be more appropriate than scoring insights made in a single session, which tend to be surface level.
In <<cite Saraiya2010>> and <<cite North2011>>, thematic categorization of insights using an open-coding method (similar to [[Grounded Theory]]) is used. In a short study with student participants and a toy dataset, insight scoring isn't an informative method - most insights are surface level, without much domain-specific inference. Categories of insight can be iterated on with domain experts. A qualitative comparison between insights generated by alternative visualizations can then be undertaken. Another alternative is to code insights using analytical taxonomies: <<cite Amar2005>>, <<cite Amar2004a>>.
!!References
<<bibliography>>
!!Class Activity
Interviewing to determine the //culture of graduate school.//
#Draw a timeline that depicts the typical progression of students through your graduate program. Include the major milestones and indicate how long each phase between milestones usually takes.
##[indicating a long phase] Why does this phase take so long / [indicating a short phase] what is this phase so short?
##Where do students in your program experience the most difficulty? For those that don't succeed in the program, at what phase do they leave the program?
##What happens for students in your program after the timeline? What are students in your program doing before the timeline starts?
#For this phase [indicating a phase] of the timeline, how do students in your graduate program spend their time? What's a typical day?
#For this phase [indicating a phase] of the timeline, where do students in your program turn for support?
##For mentoring?
##For financial support?
##What are the other forms of support here?
!!Literature Notes & Commentary
<<cite Fontana1994 bibliography:Bibliography-EPSE595>>
*Interviews in ~C-TOC interruption study and usability study: survey interviews, quantitative in nature, a clear interviewer-interviewee role - an expected set of responses, could be quantified
*Semi-structured interviews in the [[DR in the wild project|HDD-DR Ethnographic Project]]: part survey, part open-ended - mixed method approach, however interview roles were more of a master-apprentice approach - particularly when interviewing those with expertise in a specific domain, where you as an interviewer may have little to none.
<<cite Krueger2002>>
*Focus groups in ~C-TOC cultural acceptance study - moderators, note-takers, participants, stakeholders - what role did I fill (was neither a moderator or assistant moderator)? I had a different intention from the moderator (a focus on usability rather than a focus on cultural acceptance - or was it an interaction of these two factors that I cared about?) - Nor was I a participant (I was an observer waiting to hear aspects of the focus group that were relevant to my interests in the project. I did however contribute questions to the moderator
**I never saw the transcriptions of these focus groups
*Focus groups in industry - requirements analysis focus groups, inclusion of multiple users: clients, users, managers - often done over conference calls, several locations phoning in
*Krueger's focus group analysis guidelines are similar to HCI use case analysis, requirements analysis, brainstorming: flip charts, colour coding, colour paper, cutting up and coding via placement - the same analysis could take place after a series of individual interviews, observation sessions.
*Krueger says: //Be cautious of serendipitous questions - save until the end// - (the rat hole problem? going off track?)
!!References
<<bibliography>>
*re: meeting times
**more flexible by 2nd week of Nov.
**email early next week for meeting time (Thursday afternoons generally good on non-DLS days)
**possible off-campus meetings w/ TM on Fridays?
**alt: ask if TM's other meetings can be swapped
*re: breadth course requirement - ask JP for feedback / turnaround information
**systems breadth requirement: CPSC 508 (email prof for when it will be taught, pre-reqs, recommendations)
***alt: CPSC 416 distributed systems
*[[wiki|Home]] overview
*[[Literature Review]] progress
*research goals / questions
**mixed methods for infovis eval: beyond time and error
**possible in-depth qualitative field work to eval. infovis in multi-tasking settings
Links in bold blue open on the same page.
*Meeting 11:15am
*Supervisory form
*[[Literature Review]] / [[Reading|References]]:
**[[References: Read]] - papers/chapters I've read so far:
***[[Information Visualization Evaluation: Meta-Analysis|]] - the state of evaluation in ~InfoVis. Taxonomies, guidelines.
*** [[Information Visualization Evaluation: Qualitative Methods]] - effective, ineffective, and inconclusive methods.
*** [[Information Visualization Evaluation: Quantitative Methods]] - ineffective methods
*** [[Information Foraging and Sensemaking]] - debunking buzzwords, potentially spinning these as design/evaluation heuristics
**[[References: To Read (Priority)]]
***remaining [[BELIV|References: BELIV]] proceedings, esp. those in //Information Visualization// (July 2011) delayed 12 months by UBC library
**[[References: To Read]] (lower priority)
**Creativity and Cognition conference: http://dilab.gatech.edu/ccc/ (B. Shneiderman) - creativity / problem solving / insight
**to do: talk to CT about qual. methods / GT in CSCW / read Corbin & Strauss book
*Potential project ideas
**M. Sedlmair - HDD / MDS study, ethnographic study / working partnerships with research groups (local or remote)
***joint meeting w/ TM, JM?
**S. Ingram / J. Stray - use of MDS/Glimmer in journalism (hacker journalism, i.e. wikileaks info dump)
***task-switching and qualitative analysis: crisis-driven vs. curiosity-driven multi-tasking w/ vis tools
***to do: read blog: http://jonathanstray.com/
**collaborating with other research groups on/off campus, inside/outside Vancouver, those with/without existing vis. tools.
**Text summarization research (R. Ng, G. Carenini)
*Grad Student Workshops and Events, [[Graduate Pathways to Success Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]
**From Academic Researcher to Commercial Writer (2 day workshop, Oct 19-20)
**Project mgmt. workshops Oct 25 2pm (2 hour intro), Dec 6 (all day)
**Scientific/technical writing workshops Dec 1, Dec 8
*development of my fabulous wiki: [[Journal]], [[Literature Review]], [[Meeting Minutes]], [[Glossary]]
*Course breadth req.
**will sit in on 313 (ugrad systems course) next term
*December holiday travel plans
**departing Wed Dec 14, working remotely Dec 15-16, 19-23
**returning Wed Jan 4
**travel February 2012 TBD (approx 1 week coinciding w/ Reading week)
Links in bold blue open on the same page.
*Project idea: HDD ethnography project
**Meeting w/ Michael Sedlmair 10am
*Project idea: case study: text visualization for journalism
**[[3 Difficult Document-Mining Problems that Overview Wants to Solve|http://www.pbs.org/idealab/2011/10/3-difficult-document-mining-problems-that-overview-wants-to-solve297.html]] by J. Stray
**[[TheOverview Project|http://overview.ap.org/]] - they want to build [[processing|http://processing.org]] for text visualization. Lightweight, no extensive programming knowledge required. Casual use by journalists with large datasets
**Some possible field research questions in here:
>"What are the most painful things that journalists must do with document sets? What are the things they髥 to do but can嵐y? What problems exist, and which should we focus on solving? The only way to answer these questions is to talk to users."
*Project idea: UBC Text summarization research (R. Ng, G. Carenini)
Meanwhile, my ongoing [[Literature Review]] / what I've been [[Reading|References]] last week/this week:
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite Kang2010 bibliography:Bibliography>> - BELIV '10 position paper
***<<cite Elmqvist2010>> - BELIV '10 position paper
**[[Information Visualization Evaluation: Qualitative Methods]]
***<<cite Tory2005>> - expert / heuristic reviews
***<<cite Lloyd2011>> - ~VisWeek '11 ~GeoVisualization paper (Michael's recommendation) - long-term case study
**[[Information Visualization Evaluation: Quantitative Methods]]
***<<cite Chang2010>> - BELIV '10 learning-based evaluation (I like this one)
**[[Information Foraging and Sensemaking]]
***<<cite Pirolli2009>> - ch. 1 - explaining the theory
***<<cite Pirolli2009>> - ch. 9 - design heuristics for the web
**[[Evaluation in HCI: Meta-Analysis]]
***<<cite Greenberg2008>> - usability evaluation considered harmful (CHI 2008) (MUX discussion #1)
***<<cite Crabtree2009>> - ethnography considered harmful (CHI 2009)
**[[Research through Design]]: <<cite Zimmerman2010>> (MUX discussion #2)
*[[References: To Read (Priority)]]
**papers & book chapters on deck / ongoing
*[[References: To Read]] (lower priority)
**roughly 100 papers or book chapters
**now thematically categorized
**I've been broadening the scope to other HCI communities (CSCW, Creativity & Cognition, etc.)
***pinged CT for grounded evaluation in CSCW papers
Recent events:
*Distinguished Lecture on Design Engineering: //Dancing with Ambiguity: Embracing the Tension between Innovation and Decision-making in the Design Process//. Larry Leifer, Dept. of Mechanical Engineering, Stanford University [[http://cdr.stanford.edu]]
!References
<<bibliography>>
Projects:
*DR ethnography project
**Meeting w/ MS (x2) 11.11.04, 11.11.08: [[minutes|MS-11.11.04]]
**Refreshing my conceptual understanding of [[Dimensionality Reduction]]
**Reading / taking notes on project materials: notes, transcripts
***[[Cognitive Work Analysis]]
**7 of 16 user / user groups need summaries (concentrating on these, using most recent summaries as template), early summaries likely need revision
*Overview case study project: text visualization for journalism
**[[3 Difficult Document-Mining Problems that Overview Wants to Solve|http://www.pbs.org/idealab/2011/10/3-difficult-document-mining-problems-that-overview-wants-to-solve297.html]] by J. Stray
**[[TheOverview Project|http://overview.ap.org/]] - they want to build [[processing|http://processing.org]] for text visualization. Lightweight, no extensive programming knowledge required. Casual use by journalists with large datasets
**Some possible field research questions in here:
>"What are the most painful things that journalists must do with document sets? What are the things they髥 to do but can嵐y? What problems exist, and which should we focus on solving? The only way to answer these questions is to talk to users."
*[[Vismon]] project
**TBD after paper draft talk at ~InfoVis group meeting
*Project idea: UBC Text summarization research (R. Ng, G. Carenini)
Meanwhile, my ongoing [[Literature Review]] / what I've been [[Reading|References]] last week/this week:
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite vanWijk2006 bibliography:Bibliography>> (Views on Visualization)
**[[Information Visualization Evaluation: Qualitative Methods]]
***<<cite Lloyd2011>>: re-read for ~InfoVis group discussion
***<<cite Winckler2004>> - evaluation with a scenarios generated by task taxonomy
***<<cite Borkin2011>> - ~HemoVis
**[[Information Visualization: Techniques]]
***<<cite Steinberger2011>> - context preserving visual links (''~VisWeek best paper award'', and commentary)
*[[References: To Read (Priority)]]
*[[References: To Read]]
**100+ papers or book chapters
**thematically categorized
**broadening the scope to other HCI communities (CSCW, Creativity & Cognition, etc.)
***pinged CT for grounded evaluation in CSCW papers (CT away)
Upcoming:
*CHI rebuttals (this weekend / next week)
*Maneesh Agrawala visit / scheduling update
**KB demos - when/where?
*Graduation ceremony Nov. 25
*Moving early December, exact dates TBA
*Technical writing workshop Dec 1, 8 (conflict with ~InfoVis group meetings) - I may only attend Dec 1 session
*Working remotely Dec 14-16, 19-23 (in Ottawa)
*Visiting family reading week 2012 (mid Feb - TBA)
*Getting a new Laptop
!References
<<bibliography>>
11am Thursday
*Meeting scheduling for the term: week 1: flex/no meeting, week 2: all, week 3: TM, week 4: ''JM''
Projects:
*[[Vismon]] project
**[[Vismon: Research Questions]]
**R. Peterman's response
**Our position
*[[HDD-DR Ethnographic Project]]
**completed summaries for each case study
**constructing the taxonomy, reframing parts as a Q+A process
**next week: case study write-ups?
*Other potential projects:
**[[Overview Project|http://overview.ap.org/]] - data journalism - what are users actually doing?
***J. Stray visiting Vancouver in late Feb / early March
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work
***to do: brainstorm possible RPE projects around interruptions, multi-tasking - likely laboratory study
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**epistemologies: objectivism vs. constructionism vs. subjectivism
***positivism, interpretivism (symbolic interactionism, phenomenology, hermeneutics), critical inquiry
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[Reading recently|References: Read]]:
*<<cite Crotty1998 bibliography:Bibliography>>: foundations of social research (EPSE 595 text)
*<<cite Gigerenzer2007>> - gut feelings, hunches, intuition: the intelligence of the unconscious, decision-making under uncertainty
*Kosara, R: [[The State of Information Visualization, 2012|http://eagereyes.org/blog/2012/state-information-visualization-2012?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+EagerEyes+%28EagerEyes.org%29]]
[[References: To Read]]:
*<<cite Pascoe2007 bibliography:Bibliography-EPSE595>>: case study (critical inquiry) (EPSE 595 text)
*<<cite Silverman2007 bibliography:Bibliography-EPSE595>>: short, interesting, cheap book about qualitative research (EPSE 595 text)
*Creativity & Cognition, CSCW proc.
*Charmaz, K. - [[Constructing Grounded Theory|http://www.amazon.ca/Constructing-Grounded-Theory-Practical-Qualitative/dp/0761973532]] - 2hr reserve at UBC library
*D. Kahneman (2011): //Thinking Fast and Slow// (addressing the same issues as <<cite Gigerenzer2007 bibliography:Bibliography>>, however taking a different stance)
Recently:
*last week: [[GRAND workshop at SLAIS on text / social media analysis|Workshop on Text and Social Media Analysis]] (Thursday AM)
*study participant recruitment guidelines on the twiki
*CHI submission
**fixed author affiliation typo, ordering
**CHI madness preparation
***no time/place info (they already have it), no results (don't give too much away)
*MAGIC course list - qual. methods?
**qual. methods course talk to MUX
Upcoming:
*444 on call new tweak (Feb 3)
*visiting family Feb 6-14
*RPE timeline
**dependencies on user studies: participants, collaborators
**meeting next Friday 1pm
*Internship positions (Winter / Summer 2013) - e.g. Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
11am Thursday
*Meeting scheduling for the term: week 1: flex/no meeting (TM), week 2: TM +JM, week 3: (TM conflict, JM instead), week 4: (JM conflict, no meeting)
Upcoming:
*CHI submission
**registration (Mar 13) - done
**CHI madness preparation (Mar 23) - draft + feedback
**budget - hotels, flights, meals, other?
Projects:
*[[DR in the Wild|HDD-DR Ethnographic Project]]
**writing this week (focus on RW, methodology) - draft #1 by the weekend
**paper draft reading next Friday
**submit March 30
**taxonomy figure creation
**meeting w/ MS tomorrow
*[[Overview]] Project - data journalism
**[[Overview Discussion 12.03.05]]
**2 studies being run in parallel:
#post-deployment field data collection from real users regarding usage patterns, context of use: email, phone/skype - to tie in with [[EPSE 595]] final project proposal
#insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
*to do:
**generate deployment survey Qs for #1, iterate w/ JS, TM
**brainstorm methodology for #2
**project timeline and the RPE, determine supervisory committee, see [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*Other potential projects:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work - research Qs addressing context of use
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique, then possibly extending to HD to HDD/DR data
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**[[Interviewing]] - social constructivist perspective, structured, open-ended, post-modern, [[Group Interviews]]
**[[grounded theory|Grounded Evaluation]]
**[[Ethnography]] - MIT anthropologists studying marine biologists in the wild
**[[PhotoVoice|http://prezi.com/_m_lndsuctib/photovoice/]]
**[[Material & Environmental Data]]
**[[Narrative Analysis|http://prezi.com/i9c0uupsydvi/narrative-analysis/]]
**[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
**[[assignment 2|EPSE 595 Workbook 2]] participant observation, field notes
**[[assignment 3|EPSE 595 Workbook 3]] interviews
**upcoming:
***assignment 4: found data / material culture (next week)
***assignment 5: data analysis and representation, themes (Apr 4)
***final project: interpretive / critical project proposal: [[Overview]] project field interviews (Apr 11)
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[References: To Read]]:
*Hayes' Action Research in HCI (referred to by MS)
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Other upcoming:
*qual. methods course talk to MUX in April / May
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
11am Thursday
*Meetings in March: ''04/05'': flex meeting (JM), 04/12: TM +JM, 04/19: (TM), 04/26: (JM - last meeting before GRAND / CHI)
Projects:
*[[Overview]] Project
**[[Overview Discussion 12.03.05]]
**2 studies being run in parallel:
***[[Overview Deployment Field Survey]] - post-deployment field data collection from users - to tie in with [[EPSE 595]] final project proposal (Apr 11)
***insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
**to do:
***iterate on [[deployment survey Qs|Overview Deployment Field Survey]]
***brainstorm methodology for #2
***RPE: determine supervisory committee, see [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*Other potential projects:
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique, then possibly extending to HD to HDD/DR data
**CT follow-up journal paper on interruptions work - talk to CT re: commitment, level of involvement
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**Recent Topics:
**[[Organizing and Making Sense of Data]]
**[[Action Research|http://prezi.com/0lc7aiia6rlx/action-research/]]
**[[Phenomenolgy|http://prezi.com/gudzqww9jcv8/phenomenology/]]
**[[Computer Assisted Data Analysis, Data Displays]]
**[[Representing Knowledge]]
**Assignments:
***[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
***[[assignment 2|EPSE 595 Workbook 2]] participant observation, field notes
***[[assignment 3|EPSE 595 Workbook 3]] interviews
***[[assignment 4|EPSE 595 Workbook 4]] found data / material culture
***[[assignment 5|EPSE 595 Workbook 5]] preliminary data analysis
**Upcoming:
***final project: interpretive / critical project proposal: [[Overview Deployment Field Survey]] (Apr 11)
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[References: To Read]]:
*continue reading <<cite Charmaz2006>> - Grounded Theory
*Confessions of a Grounded Theory ~PhD (Furniss, CHI 2011)
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Upcoming:
*CHI practice talk at MUX, April 25
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
!!References
<<bibliography>>
11am Thursday
*Meetings in April: 04/05: flex meeting (JM), 04/12: TM +JM, 04/19: (TM), ''04/26'': (JM - last meeting before GRAND / CHI)
CHI presentation
*debrief from MUX practice talk
Projects:
*[[Overview]] Project - see [[notes from meeting w/ JS|Overview Discussion 12.03.05]]
**2 studies being run in parallel:
***1. [[Document mining in data journalism]] - post-deployment field data collection from users - [[EPSE 595]] final project proposal (submitted Apr 11) - ''awaiting feedback''
****[[interview foci and questions|Overview Deployment Field Survey]]
****User #1: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - ''still in progress as of Apr 13''
*****followed-up w/ JS - piloting, coordinating interviewer roles, interviewing AP Caracas bureau chief -'' awaiting response''
***2. insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
****possible VIVA connections via RR to UBC journalism school, Vancouver Sun
****''to do'': brainstorm methodology, ''awaiting response from JS'' re: involvement of faculty and students
**''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
**email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis: [[final research proposal|Document mining in data journalism]]
[[References: Read]]:
*<<cite Charmaz2006 bibliography:Bibliography>> - Grounded Theory ref
*<<cite North2011>> - Insight evaluation IV journal paper - comparison with benchmark task method
*<<cite O'Brien2010>>: insight-based evaluation for Gremlin: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*<<cite Endert2012>>: [[CHI 2012 on semantic interaction|http://people.cs.vt.edu/aendert/Alex_Endert/Research_files/Endert_CHI2012_Semantic_Interaction.pdf]]: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
[[References: To Read]]:
*Confessions of a Grounded Theory ~PhD (Furniss, CHI 2011)
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
Upcoming:
*CHI May 6-15
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
!!References
<<bibliography>>
11am Thursday
*Meetings in May: 05/02: TM, 05/10: (at CHI), 05/17: TM, ''05/24: JM'', 05/31: flex / as needed
[[Overview]] Project
*Exciting methodological overlap b/w the two studies: both are largely qualitative, both involve recording insight. However the two studies differ in two important ways: (1) professionals vs. students; (2) personal vs. prepared datasets (both being be real, and not toy, datasets; datasets used by Overview users in the course of their ongoing work vs. datasets we are familiar with that we give to students.
>@@color:#444bbb; ''JS'':Well the obvious choices for me might be Columbia, or another j-school in NY. Looks like I'm going to be co-teaching a course at Columbia in the fall, so that's probably most likely.@@
>
>@@color:#444bbb;Of the handful of document sets that someone has really looked at in Overview, it seems to take 6-12 to get a good set of tags for document sets in the 10k range. Perhaps we could find ways to reduce that, but it definitely seems like something that would have to be done for course credit.@@
>
>@@color:#444bbb;But, how does this relate to collecting information about use "in the wild" ala Jarrel Wade? Do we still do the same sort of interview protocol for the students vs. professional users? Are there two different study designs here?@@
>
>''MB'': there would be two study designs, however they would be similar in many respects. Some key differences:
>
>"in the wild" users ala Jarrel Wade would use Overview with their own data, while we would provide the dataset for students to work with.
>students would get both study conditions, Overview and search-only. We will rely on retrospective anecdotes from "in the wild" Overview users about prior search-only projects (if any).
>students' time would need to be constrained in order to align with their academic calendar, while we don't have as much control over how long "in the wild" users spend with Overview, as many may be working with deadlines
>
>The two studies would be similar in terms of data collection and analysis methods. In both cases we will conduct interviews with users and collect artifacts (log files, screen captures), and the analysis of this data will be similar for both studies, facilitating comparisons between students and "in-the-wild" Overview users.
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[interview foci and questions|Overview Deployment Field Survey]]
*[[spreadsheet of users|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE]]
*User #1: JW: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - 'still in progress as of May 07
>@@color:#444bbb; ''JS'':He's not quite done yet but in the home stretch, and I think there's some really interesting stuff in here.@@ (JW email)
*User #2: [[Dan Cohen|http://www.dancohen.org/]], Associate Professor in the Department of History and Art History at George Mason University and the Director of the Roy Rosenzweig Center for History and New Media.
**Fred Gibbs: email on Overview in Digital Humanities (May 4/8)
*User #3: K. Watson at ITP / NYU: [[ITP thesis archivers using Overview|http://blog.itparchive.com/post/20561066325/experimenting-with-the-associated-press-developed]]
*followed-up w/ JS - piloting, coordinating interviewer roles, interviewing AP Caracas bureau chief
**pilot interview later this week / next week
**I take ownership on questions
**AP Caracas notes - mainly usability notes
*Advertise study ("tell us what you're able to do in Overview") on the AP site (done)
*Overview is currently logging (have JS' log file, from war logs example):
**program start and shutdown
**tag creation and deletion
**adding and removing documents from tags
**view document in integrated viewer
**view document in external browser
**MDS view select, pan, zoom
**load tag file
**select node in tree view
**change tree pruning level
2. [[An Insight-Based Evaluation of Overview]] with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
*Between or Within Subjects?
>@@color:#444bbb; ''JS'': Why do you want to have each subject try both methods?@@
>
>''MB'': A few reasons. First, it's a tradeoff: a within-subjects study would require less participants but more time per participant. It will allow us to ask all participants to make comparisons between both methods; we'll be able to study whether the ordering of methods has an effect on the resulting analysis. It will also allow us to better detect particularly strong and weak students (regardless of method).
**Longitudinal ~In-The-Wild vs. ~Single-Session ~In-The-Lab?
>@@color:#444bbb; ''JS'': Sounds like little bit of a different protocol than the think-aloud approach used by previous researchers. Depends on the subjects to understand what an "insight" is and keep good notes, no? Or perhaps from the walkthrough and the tagfile/logs we could reconstruct the time of each insight?@@
>
>@@color:#444bbb;And yes, I think looking at the final stories, if any, is really interesting. It's relevant to a possibly different set of questions, regarding how much of the insight makes it into final product.@@
>
>''MB'': Due to the longitudinal aspect of this study, we won't be able to use a think-aloud approach. Instead, we'll rely on a guided retrospective walkthrough of their analysis via video chat. Re: keeping notes and understanding what constitutes insight, we can provide participants with some initial prompts or characteristics of insight. A few of the published insight studies have asked participants to list potential findings, hunches, or questions about the data before exploring a dataset; whenever an item on this list is addressed during their analysis, they would be encouraged to take note. Maintaining regular notes could also be a requirement of the course assignment.
**Open coding vs. theoretical coding? (i.e. the Amar / Stasko taxonomy)
>potentially another paper's worth of material here
**Video screen-sharing show-and-tell / insight walkthrough
Overview misc:
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: http://lensingwikipedia.cs.sfu.ca/
**TMailable: Jun 3-15; AS unavailable May 27 - Jun 7; Jun 20 - Jul 21 (window: Jun 16-19; May 23-25)
>@@color:#444bbb; ''AS'': //"Overview is really interesting: the best thing I like about it is that you have a subject expert who is integrated into your design process. I would like to discuss things from a different perspective, that of natural language processing of the underlying text, if you are interested."//@@
**this makes me curious as to who the intended end users are, what are their tasks, what else is involved in their research / analysis process. There may be an evaluation project lurking somewhere in here. I'll keep this in mind.
>@@color:#444bbb; ''AS'': //"it is an early prototype and we are rapidly developing and testing new design decisions. [...] My purpose is to make discovery of useful tuples such as <(person/place/thing) , (relationship/event) , (to person/place/thing), time, location> from large document collections. We ignore other useful characteristics of events currently."//@@
*''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*possible VIVA connections via RR to UBC journalism school, Vancouver Sun, followed-up, ''awaiting response''
*Scott (MUX): RR's VA challenge group (cog. systems ugrad students - tool/VA expertise, no domain expertise in journalism)
*compiling [[Data Journalism: Links and Literature]] (from JS, CWA)
*email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
*software:
**[[Skype Premium|http://www.skype.com/intl/en-us/prices/premium?intcmp=CS-Upsell-NarrowRight]] ($60 / annually)
**[[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]] (~20$) (purchased)
**[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595 ($39 education license) (may use for transcription only)
**[[dedoose|http://www.dedoose.com/]] - JW, CT, MH recommended (web app, $13 monthly subscription service, 15% discount if bought for 6mo/12mo), has transcription functionality (video and audio hosting surcharges): [[sign-in here|https://www.dedoose.com/App/?Version=4.2.82]]
***familiarizing myself with [[dedoose|http://www.dedoose.com/]] (adding documents, descriptors, linking documents with descriptors)
Overview for music discovery (fun little side project)
*[[scraperwiki.com|https://scraperwiki.com/profiles/mattbrehmer/]] - PHP / python web scraper
*mining the Rolling Stone Top 500 Albums of all time in Overview
*mining the last 3 months of [[Pitchfork|http://www.pitchfork.com/]] album reviews in Overview (N = ~360)
[[Graphical Inference User Studies]] Project
*[[compiled list of papers|Graphical Inference Evaluation]] citing original graphical inference paper
*''to do'': read Heer, Kosara, et al Mechanical Turk papers, some CHI papers re: crowd sourcing
[[recently read|References: Read]]:
*<<cite Eisenstein2012 bibliography:Bibliography>> - ~TopicVis text visualization from GT / CMU
*<<cite Charmaz2006>> - Grounded Theory ref (''ongoing'')
[[to read|References: To Read]]:
*Heer, J. and Shneiderman, B. (2012). Interactive Dynamics for Visual Analysis. Communications of the ACM.
*[[References: CHI2012]]
!!References
<<bibliography>>
*Meetings in Sep: TM: 09/05, TM: 09/12, ''JM: 09/20'', TM/TBD: 09/26
1. [[Overview]] Project
*[[RPE|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]] presentation Fri Sep 28 4-5pm
1a. [[Document mining in data journalism]] - post-deployment field data collection from users
*Overview Users
**keeping track of users' status in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**5 pending, 3 unknown, 3 completed (+1 pilot), 3 aborted
***pending: AP journalist w/ Paul Ryan correspondence (09/04), molecular/cellular bio researcher with paper abstracts (08/29), York U communications researcher on ~WikiLeaks (08/27)
***Jenkins (~WikiLeaks) (finally got it working after a lot of config hassle last week)
1b. [[An Insight-Based Evaluation of Overview]] with data journalism students
*study more likely to occur in Winter 2013 with [[Overview 0.1 Alpha|http://overviewproject.org]] (released today!)
*[[current course syllabus|http://www.compjournalism.com/?p=6]]
2.Task Taxonomy project
*New lit review page: [[Task Characterization: Meta Analysis]]
*lots of reading!
*seeking task / user goals taxonomies in other areas of HCI / HII / IR / library / info sciences (pinged JD) - CSCW / personalization task taxonomies could also be helpful
3. Tableau internship potential 2013 - upcoming phone/Skype call w/ J. Mackinlay (Sept)
>//"I don't think Tableau is very interested in an investment in formal evaluation methodologies because we are having tremendous success with informal evaluation. We are one of those lucky companies that enthusiastically use the software we create. Nevertheless, I would be delighted to talk to Matt."//
*emailed last 09/06 (no response so far)
4. Extending/validating <<cite Buja2009 bibliography:Bibliography>> and <<cite Wickham2010>>'s graphical inference methods
5. Upcoming:
*~VisWeek 2012 registration (done) (Oct 14 - 19) - BELIV workshop Oct 14/15 (accom, travel - done)
*need to find a venue for west coast ~VisWeek party Oct 17 (100-200 people standing, walking distance to Sheraton, no live music, 1000$ tab)
*DRITW next steps meeting (dedicating time to revisions after ~VisWeek)
*CPSC 508 (operating systems) TR 11:00-12:30
!!References
<<bibliography>>
1. Project updates
*[[Task Taxonomy|Task Characterization: Meta Analysis]]
**starting to converge
**03/13 ~VisWeek target
*Overview
**interviewed AP journo 12/5
*DRITW
**TVCG journal submission nearly there鮦oVis group review 12.19
*CPSC 508
**[[survey paper|CPSC 508: Project]] on OS evaluation methodologies
**due 12.21
2. CTOC
*TOCHI paper
*visiting student and interruption study
3. Misc
*CHI '13 SV logistics
*internship / collaboration ideas
*Upcoming holiday travel - working remote 12.14 - 12.21, xmas break 12.24 - 12.31
*Meetings in Jan: TM 01/09 11am, ''JM+TM 01/16'' (early pre-paper talk draft), TM 01/17 (pre-paper talk prep), TM 01/23 (pre-paper talk debrief), TM 01/30
1. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*I will be seeking feedback on my ''pre-paper talk'' (slides forthcoming)
*Reading recently:
**<<cite Roth2012 bibliography:Bibliography>> on [cartographic interaction primitives|Task Characterization: Meta Analysis]]
**<<cite Roth2012b>> on an empirically-derived cartographic interaction taxonomy
**<<cite Norman1988>> on seven stages of action (motivated <<cite Roth2012>>'s ''objective, operator, objective'' classification)
**<<cite Klein2006>> and <<cite Klein2006a>> on the data/frame-model of sensemaking
**<<cite Pike2009>> on the science of interaction in ~InfoVis (includes meta-taxonomy)
**<<cite Lee2012a>> on beyond WIMP interaction in ~InfoVis
2. TOCHI Review request
3. ~C-TOC interruption follow-up study w/ CJ + visiting student
*Time commitment, likely a non-trivial amount of coding; thoughts on my suggested compromise experimental design
4. rejected TOCHI "interruptions in the wild" submission
*New venue?
>''CT'' (12/13/12): //"I think we should revise it to submit to another venue. TOCHI seems too high a calibre, as Joanna pointed out, for this paper :(//"
*Replied (01/02/13) re: next steps, radio silence from CT
5. Potential research internships / collaborations summer '13:
*[[IBM research opportunities|http://ibm-research.jobs/cambridge-ma/research-summer-intern-cambridge/33144214/job/]]
*[[MSR internships|https://research.microsoft.com/apps/tools/jobs/intern.aspx]] w/ [[CUE|http://research.microsoft.com/en-us/um/redmond/groups/cue/#people]], [[VIBE|http://research.microsoft.com/en-us/um/redmond/groups/vibe/vibewebpage/#listName=peopleList&id=80389]], [[CLUES|http://research.microsoft.com/en-us/groups/clues/]]
*[[Google UX Research internship|http://www.google.com/intl/en/jobs/students/tech/internships/uscanada/user-experience-research-intern-summer-2013-north-america.html]]
*meeting w/ R. Leung 01/19 re: VA research @ SAP
*meeting w/ MS + TM on follow-up work to DRITW, cluster separation factors studies (not yet scheduled)
6. Overview
*new user story w/ Overview-web: [[The Best of Our Gun Debate: Readers Weigh In on Owning Firearms|http://www.thedailybeast.com/articles/2012/12/22/the-best-of-our-gun-debate-readers-weigh-in-on-owning-firearms.html]] - will be contacting him re: interview / logs
7. Misc.
*Selected as CHI '13 SV - Apr. 26 - May 2
**[[CHI '13 workshops|http://chi2013.acm.org/authors/call-for-participation/workshop-participants/]]
***[[Evaluation methods for creativity support environments|http://ecologylab.net/workshops/creativity/]]
***[[Many People, Many Eyes: Aggregating Influences of Visual Perception on User Interface Design|http://people.seas.harvard.edu/~reinecke/manyeyes/]]
*considering taking/sitting in on a 8-week [[Data Analysis|https://www.coursera.org/course/dataanalysis]] on Coursera starting Jan 22
*Joined some meetup groups: [[HxD Vancouver|http://www.meetup.com/HXD-Vancouver/t/wm1?rv=wm1&ec=wm1]] (first meeting next week), [[Vancouver Data Visualization|http://www.meetup.com/Vancouver-Data-Visualization/t/wm1?rv=wm1&ec=wm1]]
*attended [[Towards Personal Visual Analytics]] - guest presentation by Sheelagh Carpendale, University of Calgary (01/15/13)
*Out of town over reading week (Feb 18-22, working remotely)
!References
<<bibliography Bibliography>>
My ongoing weekly research journal:
<<list filter [tag[journal]]>>
Hello World. /%[>img[hiking in the carpathians|http://people.cs.ubc.ca/~brehmer/images/mb_tiny.jpg][http://cs.ubc.ca/~brehmer/]]%/
This is my inaugural [[Journal]] entry using [[TiddlyWiki|http://www.tiddlywiki.com/]].
I'm back from nearly a month in Europe (see evidence at right). Here is what I'm doing this week:
!!4 October
*Catching up on a few hundred emails
*Addressing administrative issues (registration, ~UPass, awards, website updates, etc.)
*Configuring my [[TiddlyWiki|http://www.tiddlywiki.com/]] at http://cs.ubc.ca/~brehmer/wiki/
*Attending meetings: LUNCH, MUX, ~InfoVis group
*Gathering some [[References]] for project brainstorming and my [[Literature Review]]
*Reading <<cite Lam2010 bibliography:Bibliography>>, making notes
!!5 October
*Configuring my [[TiddlyWiki|http://www.tiddlywiki.com/]] at http://cs.ubc.ca/~brehmer/wiki/
*Email response to prospective grad
*Attending the MUX meeting
**Research updates from Y.S. (TAMER project), V.L. (Scrolling interactions with programmable friction)
**Next week: visiting researcher presentation by J.L.
*Gathering some [[References]] for project brainstorming and my [[Literature Review]]
*Reading <<cite Lam2010 bibliography:Bibliography>>, transcribing and organizing notes on [[Multi-Level Interfaces: Meta-Analysis]]
!!6 October
*Added to ~InfoVis group mailing list
**~InfoVis group meetings Thu 15:00-16:30 in x530 (except on DLS days 14:00-15:30)
**~InfoVis group meeting discussion lead Nov. 3, Dec 15
*Reading Munzner, T. (2011) textbook ch.8 (DRAFT)
**Attribute reduction methods
*Meeting w/ TM: [[minutes|TM-11.10.06]]
*Floor Wardens meeting 12:30-13:30
*~InfoVis group meeting 15:00-16:30: [[minutes|InfoVis-11.10.06]]
!!7 October
*Reading <<cite Lam2011 bibliography:Bibliography>> and <<cite Carpendale2008 bibliography:Bibliography>>
*Making discussion notes on [[Information Visualization Evaluation]] for [[InfoVis 13 October 2011]]
!!11 October
*reading <<cite Saraiya2004 bibliography:Bibliography>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*LUNCH meeting
*reviewing JM's Discovery grant proposal + finding appropriate references
*re: prospective grad email
*reading <<cite Shneiderman2006>>, <<cite Amar2004a>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!12 October
*[[notes|Information Visualization Evaluation: Qualitative Methods]] for <<cite Amar2004a>>:
**refs added: <<cite Seo2006 bibliography:Bibliography>>, <<cite Hewett2005 bibliography:Bibliography-ToRead>>, <<cite Eick2000>>, <<cite Edmonds2005>>, <<cite Candy1997>>
*~C-TOC meeting
*visited student awards office for tuition refund cheque
*graduation ticket ordering / other details
*EPSE 595 / SOC 503 comparison
*MUX invited talk: Dr. Joel Lanir (Univ. Haifa, Israel)
**//"Examining proactiveness and choice in a location-aware mobile museum guide"//
**''free-choice learning'' - learning at one's own pace
**passive and proactive context awareness vs. menu options with varied # of choices
*reading <<cite Gonzalez2003 bibliography:Bibliography>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*reviewing <<cite Lam2011>> for ~InfoVis group discussion
!!13 October
*refs added, re-organized:
**[[References: Read]]
**[[References: To Read]]
**[[References: To Read (Priority)]]
*meeting w/ TM: [[minutes|TM-11.10.13]]
*meeting w/ JM: [[minutes|JM-11.10.13]]
*~InfoVis group meeting: [[minutes|InfoVis-11.10.13]]
*DLS: Maurice Herlihy, Professor, Brown University
**//"Muliticore, Transactions, and the Future of Distributed Computing"//
***see //Topological arrangement of asynchronous computing//
***see //Transactional memory//
***PODC: parallel algorithms, synchronization, fault tolerance (1982) -> multicore, cloud computing, peer-to-peer (2011)
***computers are no longer getting faster. if we can't exploit parallelism, why upgrade?
***synchronization the critical problem - Amdal's law
***Gartner hype cycle
***locks / threads vs. transactions, speculative, atomic
!!14 October
*document cleanup / organization
*ref added: <<cite Mayr2011 bibliography:Bibliography-ToRead>>
*graduation details - regalia renting
*workshop ~RSVPs: [[Graduate Pathways to Success Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]
**scheduling
*reading <<cite Andrews2008 bibliography:Bibliography>>, <<cite Ellis2006>>: [[notes|Information Visualization Evaluation: Meta-Analysis]], <<cite Isenberg2008>> [[notes|Information Visualization Evaluation: Qualitative Methods]]
*helping JM w/ interruptions refs for discovery grant application
!!17 October
*GRAND reporting
*Reading:
**<<cite Pirolli2005 bibliography:Bibliography>>: [[notes|Information Foraging and Sensemaking]]
**<<cite Sedlmair2011>>: [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Scholtz2010>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Seo2006>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Trafton2000>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!18 October
*completing notes from yesterday (see above)
*retrieving papers from references of above papers
*looking for Pirolli's information analyst paper; access to Information Visualization journal issue Jul '11
*LUNCH
*Reading:
**<<cite Plaisant2004>>: [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Isenberg2008a>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Saraiya2006>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Mayr2010>>
**<<cite Saraiya2010>>
!!19 October
*GPS Event: From Academic Researcher to Commercial Writer 9am-4pm
!!20 October
*GPS Event: From Academic Researcher to Commercial Writer 9am-4pm
*meeting w/ TM: [[minutes|TM-11.10.20]]
!!21 October - 22 October
*(at home)
*website updates
*reading
**<<cite Gonzalez2003a>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Valiati2006>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Perer2009>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Chen2000a>>: [[notes|Information Visualization Evaluation: Meta-Analysis]]
*booked Dec. holiday flights:
**departing Wed Dec 14, working remotely Dec 15-16, 19-23
**returning Wed Jan 4
!!24 October
*wiki config / plugins (resolving syncing issues from over the weekend).
*notes:
**<<cite Mayr2010 bibliography:Bibliography>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Saraiya2010>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*adding refs to [[References: To Read]]
*reading:
**<<cite Zuk2006>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Sutcliffe2000>>: [[notes|Information Visualization Evaluation: Quantitative Methods]]
**<<cite Thomas2005>> - //Illuminating the Path// ch. 2: [[notes|Information Foraging and Sensemaking]]
!!25 October
*downloading remaining BELIV papers: [[References: BELIV]]
*reading:
**<<cite Thomas2005>> - //Illuminating the Path// ch. 6: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*doctor's appt. 10:45
*LUNCH meeting
*GPS Event: Project mgmt. 2-4pm
*reading (cont.)
!!26 October
*notes for above readings
*IMAGER pizza social
**HCI - Idin Karuei (~PhD) and Oliver Schneider (~MSc) (~MacLean): "A Frequency Based Gait Measurement Algorithm for Mobile Phones"
**Graphics - Matthias Hullin (PDF; Heidrich): "Bixels"
*no MUX meeting
*reading BELIV '08 papers:
**<<cite Yi2008>>: [[notes|Information Foraging and Sensemaking]]
**<<cite Faisal2008>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Tory2008>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Valiati2008>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!27 October
*notes for above readings
*meeting w/ JM: [[minutes|JM-11.10.28]]
*wrote [[Academic Researcher to Commercial Writer: Workshop Notes]]
!!28 October
*reading BELIV '10 papers:
**<<cite Elmqvist2010>> - position paper (mutually linked studies): [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Kang2010>> - position paper (pragmatic challenges): [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Chang2010>> - learning-based evaluation: [[notes|Information Visualization Evaluation: Quantitative Methods]]
*and:
**<<cite Tory2005>> - expert / [[Heuristic Evaluation]]: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*UBC School of Music Contemporary Players noon concert
*notes for above papers
*reading:
**UBC/CS/LCI text summarization group web page
**http://jonathanstray.com/, http://overview.ap.org/
!!31 October
*reading:
**<<cite Pirolli2009 bibliography:Bibliography>> - ch. 1: [[Information Foraging Theory / Framework and Method|Information Foraging and Sensemaking]]
**<<cite Lloyd2011>> ~VisWeek ~GeoVisualization paper (Michael's recommendation)
*notes for above
*pinged CT for grounded evaluation in CSCW papers
!!01 November
*reading:
**[[3 Difficult Document-Mining Problems that Overview Wants to Solve|http://www.pbs.org/idealab/2011/10/3-difficult-document-mining-problems-that-overview-wants-to-solve297.html]] by [[J. Stray|http://jonathanstray.com/]]
**[[TheOverview Project|http://overview.ap.org/]] - they want to build [[processing|http://processing.org]] for text visualization. Lightweight, no extensive programming knowledge required. Casual use by journalists with large datasets
***Goals and use cases of the project:
>"What are the most painful things that journalists must do with document sets? What are the things they髥 to do but can嵐y? What problems exist, and which should we focus on solving? The only way to answer these questions is to talk to users."
>[...]"Users should be able to write and use new pieces. That襠only way we can build a system that쥸ible and extensible enough to power many years worth of exploration in this space."
>[...]"Overview is intended to be an extensible, evolving open-source system supported by a diverse community of journalists, volunteers, scholars, governments and other users who need to understand very large sets of documents of public interest 堄rupal of interactive data visualization."
*watching:
<html><iframe src="http://player.vimeo.com/video/20450035?title=0&byline=0&portrait=0" width="400" height="300" frameborder="0" webkitAllowFullScreen allowFullScreen></iframe</html>
*learning how to use [[processing|http://processing.org]]
*reading:
**MUX papers: <<cite Greenberg2008>>, <<cite Zimmerman2010>>
*LUNCH meeting
**GRAND reporting
*Distinguished Lecture on Design Engineering: //Dancing with Ambiguity: Embracing the Tension between Innovation and Decision-making in the Design Process//. Larry Leifer, Dept. of Mechanical Engineering, Stanford University [[http://cdr.stanford.edu]]
**guest lecture given to 4th yr. ugrad design engineering course. HCI researchers are wrong audience (preaching to the choir)
***a question was asked: "just how important are people in these solutions?"
**//Ambiguity// is hunting in the future. Prototyping adds to ambiguity.
**researchers need to be adaptive, switching between in-depth knowledge and design thinking behaviour, teams need a mix of divergent and convergent thinkers, ask questions, and use creative language
**a descriptive, rather than prescriptive talk - highlighting projects from his design course that have excelled
!!02 November
*mining the [[Creativity & Cognition (C&C)|http://dilab.gatech.edu/ccc/]] proceedings
*reading <<cite Crabtree2009>> : [[notes|Evaluation in HCI: Meta-Analysis]]
*MUX meeting
**discussion #1: <<cite Greenberg2008>> (Hasti will lead the discussion): [[notes|Evaluation in HCI: Meta-Analysis]]
**discussion #2: <<cite Zimmerman2010>> (Charlotte will lead the discussion): [[notes|Research through Design]]
*reading:
**<<cite Pirolli2009>> ch. 9: Design Heuristics, Engineering Models, and Applications (web design, browsers, search engine results)
**<<cite Winckler2004>> - task/scenario taxonomy for evaluation: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!03 November
*categorizing [[References: To Read]]
*meeting w/ TM + MS (HDD / DR ethnography project): [[minutes|TM-MS-11.11.03]]
*meeting w/ JM: [[minutes|JM-11.11.03]]
*mining the ~VisWeek proceedings: [[References: VisWeek2011]]
*~InfoVis Group meeting: minutes - ~VisWeek debrief session
**relevant/interesting papers from [[References: VisWeek2011]]
!!04 November
*reading:
**<<cite Steinberger2011>> (~VisWeek best paper): [[Stephen Few's blog post|http://www.perceptualedge.com/blog/?p=1090]], and [[response|http://alexlexvisual.blogspot.com/2011/10/context-preserving-visual-links.html]]
**<<cite Borkin2011>> (~VisWeek paper, recommended): [[Stephen Few's blog post |http://www.perceptualedge.com/blog/?p=1090]]
*meeting w/ MS 2pm re: HDD / DR ethnography project: [[minutes|MS-11.11.04]]
*UDLS
!!05 November
*reading:
**<<cite vanWijk2006>> (Views on Visualization)
!!07 November
*notes for:
**<<cite vanWijk2006 bibliography:Bibliography>> (Views on Visualization): [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Winckler2004>> - evaluation with a scenarios generated by task taxonomy: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*reading:
**[[The Most Interesting Papers at the Infovis Conference (VisWeek 2011)|http://infosthetics.com/archives/2011/11/most_interesting_papers_at_infovis_visweek_2011.html]]
**[[VisWeek updates by J堃ukier: Day 5|http://www.visualisingdata.com/index.php/2011/10/visweek-updates-by-jerome-cukier-day-5/]]
**[[Stephen Few⩴icism of information visualisation research|http://www.visualisingdata.com/index.php/2011/10/stephen-fews-criticism-of-information-visualisation-research/]]
*reading: notes, summaries, audio, transcripts re: DR ethnography project
**notes, assumptions, technical information, summary spreadsheet
*reading: [[Cognitive Work Analysis|http://cel.mie.utoronto.ca/research/frameworks/cwa.htm#skip-nav-target]] (Cognitive Engineering Laboratory at the University of Toronto): [[Cognitive Work Analysis]] (local)
!!08 November
*SVN issues
*reading:
**<<cite Buja2002>> - Vis. methods for MDS
*LUNCH meeting
**GRAND reporting
**new laptop?
*scheduling M.Agrawala.'s visit 11.17
**conflicts w/ JM meeting 11.17
**~InfoVis group meetings
*meeting w/ MS re: SVN/FTP setup
!!09 November
*reading/notes on [[Dimensionality Reduction]]
*~C-TOC meeting @ Max Walters room, Koerner Pavilion, UBC Hospital
*no MUX meeting
*reading:
**<<cite Holbrey2006 bibliography>> - Column Reduction techniques
**<<cite Lloyd2011>>: re-read for ~InfoVis group discussion
*reading case study notes/summaries
!!10 November
*meeting w/ TM: [[minutes|TM-11.11.10]]
*meeting w/ JM: [[minutes|JM-11.11.10]]
*meting w/ MS 2pm: [[minutes|MS-11.11.10]]
*~InfoVis group meeting
**discussing <<cite Lloyd2011>>
*ordered MB Air
!!11 November
*(Remembrance Day)
*reading DR references
**<<cite Ingram2010>>: ~DimStiller
**<<cite Tenenbaum2000>>: Isomap
*reading ethno/DR case studies
**~GenomeDX: verifying clusters, DR with noisy data, combining genomic and clinical data: need vis during DR steps; data possibly ''indifferentiated''
!!12 November
*CHI reviews are in
!!13 November
*CHI rebuttal planning
!!14 November
*scheduling M. Agrawala's visit
*settling apt. issues
*CHI rebuttal writing
*reading:
**<<cite Bertini2011 bibliography:Bibliography>>: ~VisWeek quality metrics paper
!!15 November
*notes for <<cite Bertini2011>>: [[notes|Information Visualization Evaluation: Meta-Analysis]]
*~GenomeDX SPLOM vs. 3D SP feedback
*LUNCH meeting
**CHI review recaps
*[[HDD-DR Ethnographic Project]]: case study review (HL)
**CHI 2011 submission, meeting notes (MS, TM), email correspondence
!!16 November
*CHI rebuttal 2nd iteration (JM's feedback)
*[[HDD-DR Ethnographic Project]]: case study review (HL) (cont.)
*lunch w/ M. Terry (12:30-1:30)
*MUX meeting
**guest lecture by M. Terry ([[HCI lab, U of Waterloo|http://hci.uwaterloo.ca/]])
***//Google Knows What You Hate About Your Kindle//
***CUTS: Characterizing Usability Through Search (CHI 2011) - A. Fournier
****Query intent taxonomy
***Query Feature Graphs (UIST 2011) - A. Fournier
***QAP: ~Question-Answer Protocol (Clarke et. al. 2000)
*reading:
**<<cite Springmeyer1992>> - scientific data analysis taxonomy
!!17 November
*M. Terry in CPSC 344/544
*meeting w/ TM: [[minutes|TM-11.11.17]]
**[[classification vs. clustering|http://www.broadinstitute.org/annotation/winter_course_2006/index_files/Classification_and_Clustering.ppt]] slide deck
**[[Correspondence Analysis]]
*chat/demo session with M. Agrawala (12:30-1)
*Lunch w/ M. Agrawala (1-2pm)
*[[HDD-DR Ethnographic Project]]: case study summarizing (HL)
*meeting w/ JM re: CHI rebuttal
*DLS: M. Agrawala
**Video puppetry + ~NapkinVis for Vis prototyping? Map to data sources?
*CHI rebuttal v3
!!18 November
*CHI rebuttal submitted
*~InfoVis group meeting (10:30-12:15):
**SI's pre-paper talk: Snappy
*[[HDD-DR Ethnographic Project]]: case study summarizing (HL)
!!21 November
*[[HDD-DR Ethnographic Project]]: case study review (DH)
**compiling a list of terminology I don't understand
*reading: <<cite Hullman2011a bibliography:Bibliography>>: Visual Difficulties (~VisWeek 2011 honourable mention)
*MAGIC Demo Day (4-7, Forestry Sciences Centre #1221)
!!22 November
*[[notes|Information Visualization Evaluation: Meta-Analysis]] for above paper, reading S. Few commentary
*LUNCH meeting
*meeting w/ MS re: [[HDD-DR Ethnographic Project]]
**DH summary review
**to interview KA tomorrow 3PM, sent 2 papers
**sent JB's CHI submission
*[[HDD-DR Ethnographic Project]]: case study summary for (DH)
!!23 November
*[[HDD-DR Ethnographic Project]]
**skimming KA's papers
*~C-TOC meeting (ICCS x530)
*IMAGER Pizza Social
**Discussion:
>//"Today we are capable of modeling and rendering nearly everything we can think of. 3D graphics hardware has now been a commodity item for many years. Be it resolved that computer graphics is losing its "mojo" because the most significant problems have been solved."//
*[[HDD-DR Ethnographic Project]]: interview w/ KA
*reading MS' paper draft for ~InfoVis meeting
!!24 November
*meeting w/ TM: [[minutes|TM-11.11.24]]
*meeting w/ JM: [[minutes|TM-11.11.24]]
**to do: write CHI rebuttal writing guide on MUX twiki
*revising [[HDD-DR Ethnographic Project]] case study summary for (DH)
*reading MS' paper draft for ~InfoVis meeting
*~InfoVis group meeting: MS paper draft
!!25 November
*M.Sc convocation ceremony (Chan Centre)
!!28 November
*reading [[Vismon]] paper draft for ~InfoVis group meeting
*~InfoVis group meeting: discussing [[Vismon]] paper draft
*scheduling term 2 week-to-week
*CSCW grounded eval papers list sent by CT
**Added [[CSCW Evaluation papers|Bibliography-ToRead-CSCWEval]] to [[References: To Read]]
*[[HDD-DR Ethnographic Project]]: case study summary (KA)
!!29 November
*feedback on CT's 344/544 lecture re: domestic interruption
*reading:
**<<cite Andre2009 bibliography:Bibliography>> - designing for (un)serendipity: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*LUNCH meeting
*[[HDD-DR Ethnographic Project]]: case study summary (Sareh)
**finding additional references re: mathematical modelling of biological aggregate movement
**reading notes, abstracts, related literature
!!30 November
*[[HDD-DR Ethnographic Project]]: case study summary (Sareh)
**sent email questions to MS for clarification, summary draft done
*Participating in ZFF's study 11-11:45
*MUX meeting: MR's thesis presentation
*reading:
**<<cite Kraut1988>>: interview, survey, archival study of informal communication in research collaboration: [[notes|Evaluation in HCI: Meta-Analysis]]
!!01 December
*(moving errands)
*on waiting list for GSS workshop on technical writing
*[[HDD-DR Ethnographic Project]]: case study summary (Sareh)
**responses from MS for clarification, summary draft revision
**reading SB's Paraglide paper draft, Sareh's M.Sc thesis
*DLS: //Computational Approaches For Large Scale Inverse Problems For Image Reconstruction// - James Nagy, Emory (did not attend)
*reading:
**<<cite Nardi2000>> - outeraction IM use - ethnographic study (CSCW): [[notes|Evaluation in HCI: Meta-Analysis]]
**interactions (Dec 11 issue)
**<<cite Jennings2011>> - C&C methodology paper for measuring exploratory visual creativity
*follow-up email to CT: CSCW refs on qual. eval. methodology
!!02 December
*(moving errands)
*~InfoVis group meeting: SI ~MoDisco paper draft (postponed)
*reading:
**<<cite North2006>> - measuring insight
*[[notes|Evaluation in HCI: Meta-Analysis]] for <<cite Jennings2011>>, [[notes|Information Visualization Evaluation: Meta-Analysis]] for <<cite North2006>>
*[[HDD-DR Ethnographic Project]]: case study review (JW)
!!05 December
*gathering refs from <<cite Jennings2011 bibliography:Bibliography>> - visual creativity
*[[HDD-DR Ethnographic Project]]: case study review (JW) - prelim. summary written
!!06 December
*reading [[MoDisco]] paper draft for ~InfoVis group meeting
*[[MoDisco]] paper draft review: ~InfoVis group meeting
*LUNCH meeting (cancelled/optional)
*brainstorming: on measuring [foo], where foo = insight/serendipity (+ gut feelings) + creativity + learning, and foo is needed for sensemaking / scientific discovery / visual analytics
*reading:
**<<cite Klahr1999>> - on Scientific Discovery
!!07 December
*reading:
**<<cite Klahr1999>> - on Scientific Discovery (cont.): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*no MUX meeting this week
*brainstorming: on measuring [foo], where foo = insight/serendipity (+ gut feelings) + creativity + learning, and foo is needed for sensemaking / scientific discovery / visual analytics
!!08 December
*meeting w/ TM + JM: [[minutes|TM-JM-11.12.08]]
*setting up MB Air - config
!!09 December
*setting up MB Air - config, ordered iWork '09
*set up SVN for [[HDD-DR Ethnographic Project]] on MB Air
*(moving)
!!12 December
*CHI paper accepted!
*setting up MB Air - config - VPN issues / calling IT help desk
*reimbursement form for iWork
*[[HDD-DR Ethnographic Project]]: SVN, case study summary / table formatting
*CPSC 533C ~InfoVis project presentations
*JP sez: ~PhD breadth form approved
*apt. cleanup
!!13 December
*EPSE 595 syllabus sent: textbook + reading list
*Library to retrieve Charmaz grounded theory book: on 2hr reserve - to buy a copy?
*attended seminar on computational population biology: Tanya ~Berger-Wolf (U. Illinois, Chicago): [[Computational Insights into Population Biology|http://compbio.cs.uic.edu/]]
*LUNCH meeting
*reading CHI review (2nd round)
*move-out inspection 3pm
*installed ~iWork '09
!!14 December
*[[HDD-DR Ethnographic Project]]: MS sent J. Buettgen note translations, to meet next Monday 10am EST
**reading note translation
*flight to Ottawa
*reading:
**<<cite Chang2009 bibliography:Bibliography>> - intelligence analysis process: long. study: [[notes|Information Visualization Evaluation: Qualitative Methods]]
**<<cite Kang2011>> VAST 2011 short note on two types of insight: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 1
*MUX meeting this week: YS's M.Sc presentation
!!15 December
*meeting w/ TM: [[minutes|TM-11.12.15]] - discussing J. Westbrook case study for [[HDD-DR Ethnographic Project]]
*[[HDD-DR Ethnographic Project]]: revising summary for J. Westbrook, reading notes / writing summary for J. Buettgen
*reading:
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 2
!!16 December
*[[HDD-DR Ethnographic Project]]: reading notes / gathering refs/ writing summary for A. Saad (SFU)
*gathering refs from [[Vismon]] paper draft
*reading:
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 3
!!19 December
*[[HDD-DR Ethnographic Project]]: meeting w/ MS: [[minutes|MS-11.12.19]]
**adding SB's Paraglide paper: AS's use case
**JW use case instance / data characterization
*added refs from <<cite Klahr1999 bibliography:Bibliography>>: on scientific discovery
*reading:
**[[Vismon|Vismon Project]] refs
***<<cite Peterman2009>> - future of fisheries research
*CHI paper revisions (results summary)
**imported to Numbers '09
!!20 December
*CHI paper revisions (results summary)
**attempting to do some charting in Numbers '09
**pondering use of R
**a workable solution in Numbers '09
!!21 December
*reading:
**[[Vismon|Vismon Project]] paper draft re-read
**<<cite Gigerenzer2007 bibliography:Bibliography>> book: Gut Feelings ch. 4
*downloading & playing w/ [[Vismon|Vismon Project]], tutorial completed
*added more [[Vismon refs|References: Vismon]]
*[[HDD-DR Ethnographic Project]]: started filling in summary table
!!22 December
*[[HDD-DR Ethnographic Project]]: summary table
!!23 December
*(cottage)
!!30 December
*[[HDD-DR Ethnographic Project]]: summary table
*reading:
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 5
!!02 January
*(Toronto)
*[[HDD-DR Ethnographic Project]]: summary table completed
*reading:
**<<cite Gigerenzer2007 bibliography:Bibliography>> book: Gut Feelings ch. 6, 7
!!03 January
*(Toronto)
*reading:
**<<cite Matusik2003>>: data-driven reflectance model / BRDF DR paper
*[[HDD-DR Ethnographic Project]]: added <<cite Matusik2003>> to summary table
!!04 January
*7AM flight to Vancouver
*reading:
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 8
*CPSC 313 DMP 110 9am-10am (missed due to flight timing)
*EPSE 595 at SCRF 204A 1-4PM - lecture based, several shorter assignments, many techniques covered, data coding, representations
*course shopping research / schedule
*selecting a paper for ~InfoVis group meeting
!!05 January
*[[CPSC 538W|http://www.cs.ubc.ca/~andy/538w-2012/]] at ICCS 206 9:30-11 (cancelled, first class Tues, Jan 10)
*meeting w/ TM + JM: [[minutes|TM-12.01.05]] (cancelled, JM + TM unavailable)
*reading:
**<<cite Rodgers2011>> for ~InfoVis group meeting: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*SOCI 503 at UCLL 101 2-5PM (heavy on methodological frameworks / positioning your research within a framework, rather than techniques/methods, taught from feminist perspective) seminar based, possible to audit, may not be entirely relevant to our field
!!06 January
*CPSC 313 DMP 110 9am-10am
*~InfoVis group meeting 11am - 12:30 ICCS x530
**paper to read: Rodgers, J. and Bartram, L. (2011). Exploring Ambient and Artistic Visualization for Residential Energy Use Feedback. IEEE Transactions on Visualization and Computer Graphics 17, 2489-2497.
**to read: Wickham graphical inference for ~InfoVis
*CHI paper camera-ready edits
*Library / Crotty book
!!09 January
*reading:
**<<cite Crotty1998 bibliography:Bibliography>> book: foundations of social research ch. 1
*CPSC 313 DMP 110 9am-10am
*meeting w/ TM + MS over Skype
**[[HDD-DR Ethnographic Project]] update + timeline
***to do: revise / refine / iterate on taxonomy (from data-centric to task-centric), case studies (find corner cases)
*CPSC 538E: Computer Architecture (topics in systems breadth course) 12:30-2pm
*scheduling + organizing
*[[HDD-DR Ethnographic Project]] taxonomy brainstorming
**<<cite Gigerenzer2007>> book: Gut Feelings ch. 8
**<<cite Crotty1998>> book: foundations of social research ch. 2
!!10 January
*[[HDD-DR Ethnographic Project]] taxonomizing + brainstorming
**summary for J. Stray
*reading:
**<<cite Crotty1998>> book: foundations of social research ch. 2,3
*CHI camera-ready submission revisions based on JM's commentary
!!11 January
*CPSC 313 DMP 110 9am-10am
*~C-TOC meeting 11:00-12:30 @ Koerner pavilion
*[[EPSE 595]] qualitative methods at SCRF 204A 1-4pm
*[[HDD-DR Ethnographic Project]] taxonomizing + brainstorming - sent ideas to MS
!!12 January
*meeting w/ TM + JM: [[minutes|TM-JM-12.01.12]]
*meeting w/ JM re: CHI submission
*CHI camera-ready submission finalizing + submit
*reading <<cite Wickham2011>> for ~InfoVis group meeting
!!13 January
*CHI revisions due
*CPSC 313 DMP 110 9am-10am
*meeting w/ MS: 10am - 11am
**current state of the taxonomy
**next steps: MS to continue brainstorm, token back to MB 12.01.16, MB to consider case studies
*~InfoVis group meeting 11am - 12:30 ICCS x530
**paper to read: <<cite Wickham2011>>: Product plots.. IEEE Transactions on Visualization and Computer Graphics 17, 2223-30. (see [[References: VisWeek2011]])
*CPSC 313 A1, studying
!!16 January
*reading:
**<<cite Crotty1998 bibliography:Bibliography>> book: foundations of social research ch. 3
*CPSC 313 DMP 110 9am-10am
*meeting w/ TM + MS
**[[HDD-DR Ethnographic Project]] update + timeline
***to do: revise / refine / iterate on taxonomy (from data-centric to task-centric), case studies (find corner cases)
*[[Vismon]] project: logistical and [[research questions|Vismon: Research Questions]]
**<<cite Crotty1998>> book: foundations of social research ch. 4
!!17 January
*Sage publications gives free access to [[Information Visualization|http://ivi.sagepub.com/content/current]] journal until Feb 15 (@benbendc)
*[[HDD-DR Ethnographic Project]]
**reading:
***email thread re: classes and clusters
***http://thesocialcanvas.blogspot.com/2011/03/classification-vs-clustering.html
***http://www.coli.uni-saarland.de/~crocker/Teaching/Connectionist/lecture13_4up.pdf
***<<cite Rubinov2006>> - Classes and clusters in data analysis
**taxonomy brainstorming
*reading:
**<<cite Crotty1998>> book: foundations of social research ch. 5
*EPSE 595 journalling
!!18 January
*CPSC 313 DMP 110 9am-10am
*CPSC 313 studying
*[[EPSE 595]] qualitative methods at SCRF 204A 1-4pm
*meeting w/ TM 4pm: [[minutes|TM-12.01.18]]
!!19 January
*[[Workshop on Text and Social Media Analysis]] @ SLAIS, 9AM - 12PM
**Dr. Stuart Shulman, President and CEO of Texifter and Founding Director of the Qualitative Data Analysis Program (QDAP)
**Irving K. Barber Centre, School of Library, Archival and Information Studies, Room 458 (4th Floor)
*reply from RP re: [[Vismon]] project, in response to [[Vismon: Research Questions]]
**SFU, Michigan State (MJ), U. Guelph (Erie / Great Lakes), Canada SDFO in Nanaimo, ADF&G (Alaska Dept. of Fish and Game)
**Most Qs to be answered by MJ (Michigan State)
>//It is not a simulation tool in the sense of being a software package with which simulations are run. Instead, it is merely an interface between a user and the output of results from simulations that have already been run//
*Iterating on [[Vismon: Research Questions]]
*CPSC 313 studying (A1 completed)
!!20 January
*CPSC 313 DMP 110 9am-10am
*meeting w/ MS: 10am - 11am (cancelled by MS)
*travel research + booking
*meeting w/ TM re: [[Vismon]] project
*follow-up email to RP, Vismon people
*pilot survey study for MH
*CHI formatting edits
*CPSC 313, studying
!!23 January
*CPSC 313 DMP 110 Test #1 (did not attend)
*[[Study recruitment resources page on the IMAGER twiki|https://bugs.cs.ubc.ca/cgi-bin/twiki/view/Imager/HCIStudyParticipantRecruitingResources]]
*meeting w/ TM + MS
**[[HDD-DR Ethnographic Project]] - MB and MS to fill in boxes w/ case studies
*LUNCH meeting
*reading:
**<<cite Crotty1998 bibliography:Bibliography>> book: foundations of social research ch. 6-7
*EPSE 595 journalling
!!24 January
*reading:
**Kosara, R: [[The State of Information Visualization, 2012|http://eagereyes.org/blog/2012/state-information-visualization-2012?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+EagerEyes+%28EagerEyes.org%29]]
>// I don᮴ to end up with a situation like in human-computer interaction, where all the constructive work is done in industry and all academics seem to do is study what others build.//
*[[HDD-DR Ethnographic Project]]
!!25 January
*CPSC 313 DMP 110 9am-10am
*EPSE 595 journalling
*~C-TOC meeting (cancelled)
*[[EPSE 595]] qualitative methods at SCRF 204A 1-4pm
*[[HDD-DR Ethnographic Project]]
!!26 January
*meeting w/ JM 11am: [[minutes|JM-12.01.26]]
*CPSC 313 studying
!!27 January
*CPSC 313 DMP 110 9am-10am
*meeting w/ MS: 10am - 11am
*~InfoVis group meeting cancelled / postponed
*CPSC 313, studying
!!30 January
*CPSC 313 DMP 110
*[[HDD-DR Ethnographic Project]] - skimming GM papers
*meeting w/ TM + MS
**[[HDD-DR Ethnographic Project]]
*LUNCH meeting
*UBC ICICS open house Mar 7th - demo summary
**[[HDD-DR Ethnographic Project]] - MB and MS to iterate, resolve differences in Q&A table
*reading:
**Pascoe 2007 for [[EPSE 595]]
!!31 January
*[[EPSE 595]] journalling
*[[HDD-DR Ethnographic Project]] - MB and MS to iterate, resolve differences in Q&A table, case studies
*RPE project brainstorming - emailed TM and JM
!!01 February
*CPSC 313 DMP 110 9am-10am
*[[EPSE 595]] qualitative methods at SCRF 204A 1-4pm
*[[HDD-DR Ethnographic Project]] - reviewing MS's notes/slides
*CPSC 313 A2
*re: RPE project brainstorming - received feedback from TM, JM
!!02 February
*CPSC 313 studying
*[[EPSE 595]] journal editing
*SI: distance matrix users from [[HDD-DR Ethnographic Project]]
*fixing typo in CHI paper author block
*[[HDD-DR Ethnographic Project]] - case study possibles
*reading
**<<cite Wickham2010 bibliography:Bibliography>> - graphical inference for infovis: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*reading for ~InfoVis group session
!!03 February
*CPSC 313 DMP 110 9am-10am
*meeting w/ MS: 10am - 11am
*~InfoVis group meeting: Relex paper draft
*RPE info session 1-2pm
*CPSC 313, studying
!!06 February - 14 February
*(in Florida visiting relatives)
*reading
**<<cite Newell1985 bibliography:Bibliography>> - prospects for psychological science in HCI
*assigned 3 GRAND reviews
*[[EPSE 595]]
**reading <<cite Silverman2007>>, Freeman and Mathison (2009), ~Tri-Council Ethics Report (2010)
**[[journalling|EPSE 595]]
**completing workbook #1
*CPSC 313
**studying (test #2)
**tutorial #5, 6
**assignment #3
**lectures: completing unit 2, beginning unit 3
!!15 February
*CPSC 313 lecture 9-10am DMP 110
*EPSE 595 lecture 1-4pm SCF 204A
**reading <<cite Becker1957>>: participant observation
**[[Workbook 1|EPSE 595 Workbook 1]]
!!16 February
*meeting w/ TM, JM: [[minutes|TM-JM-12.02.16]]
*demo for J. Birnholtz
*lunch w/ J. Birnholtz and grads
*reading Hayes, G. R. (2011) on Action Research in HCI
*DLS - J. Birnholtz - Butler Lies
!!17 February
*CPSC 313 lecture 9-10am DMP 110
*meeting w/ MS re: [[HDD-DR Ethnographic Project]]
**pre-paper talk next week
**taxonomy: task / data / user / DR
**Gaps: groups of dims, back-projections, guidance
**Implications / Discussion: gaps and guidance
**methods: discount grounded theory
**to do:
***evocative use case names
*~InfoVis group meeting: MM pre-paper talk: ~InSite
**interaction techniques for graphs
**see Johnson, Potter (Vismon ref)
*~MoDisco meeting w/ TM, SI
**to do:
***read final submission, reviews, read Yang, OST (3D landscape) papers
***generate questions for JS re: tags, clusters, tasks, motivation
*[[HDD-DR Ethnographic Project]]
**re-arrange columns in Q&A table to match current taxonomy structure
**added <<cite Matusik2003>>
!!References
<<bibliography>>
!!(Reading Week)
*CPSC 313
**test #2
**tutorial #5, 6
**assignment #3, 4
**lectures: completing unit 2, unit 3
*[[EPSE 595]]
**reading Fontana, Krueger references
**[[journalling|EPSE 595]]
**workbook #2
*GRAND reviews x3
*~MoDisco
**to do: read final submission, reviews, read Yang, OST (3D landscape) papers,
**to do: generate questions for JS re: tags, clusters, tasks, motivation
*Graphical Inference for ~InfoVis: research, who's cited, etc.
!!20 February
*withdrew ~C-TOC demo from Haptics Symposium open house (low relevance)
*sent follow-up email to CHI submission coordinator re: typo in CHI submission address block
*[[HDD-DR Ethnographic Project]]
**evocative use case names
**skimming DR papers MS listed in Feb 18 email
*no LUNCH meeting
*meeting w/ MS + TM: [[HDD-DR Ethnographic Project]] 12:00 - 5:30
**pre-paper talk
**Q&A table validation
**literature
*GRAND reviews
!!21 February
*CPSC 313 catch-up
**test #2
**tutorial #5
**assignment #3
**lectures: completing unit 2
!!22 February
*[[HDD-DR Ethnographic Project]]
**Reveret, Bronstein, Tenenbaum papers - skim/read, add to Q&A table
**preliminary data analysis
*~C-TOC meeting
**updates from CJ
**TAM model: technology acceptance model for cross-cultural acceptance of novel technologies
**study roadmap
*no MUX meeting
*meeting w/ MS + TM re: [[HDD-DR Ethnographic Project]] - 2:00 ->
*[[EPSE 595]] workbook #2
!!23 February
*CPSC 313 catch-up
**assignment #3
**assignment #4
**tutorial #6
**lectures: unit 3
*[[HDD-DR Ethnographic Project]]
**Reveret, Bronstein, Tenenbaum papers - skim/read, add to Q&A table
**use case splits, row/column re-organization
**preliminary data analysis
!!24 February
*[[HDD-DR Ethnographic Project]]
**reviewing MS's pre-paper talk slide deck
**[[Analytical Induction|http://www.youtube.com/watch?v=SizaG3KKAp4]]
**<<cite Ratcliff bibliography:Bibliography>> on [[Analytical Induction]]
***use case splits, row/column re-organization
**preliminary data analysis
*~InfoVis group meeting: [[HDD-DR Ethnographic Project]] pre-paper talk
*GRAND Reviews
*[[EPSE 595]] workbook #2
!!References
<<bibliography>>
!!27 February
*CPSC 313 9-10am DMP 110
*[[HDD-DR Ethnographic Project]]
*emailing EPSE 595 prof re: [[Analytical Induction]]
**Q&A table validation
**literature
*no LUNCH meeting
*pilot session w/ DT in w204
*taxes and finances
*reviewing journal paper draft for CT, RL, SH
!!28 February
*[[HDD-DR Ethnographic Project]]
**QA table data analysis
**DR literature review / ref gathering: [[References: DR]]
*LUNCH meeting 12:30
*[[EPSE 595]]
**reading Fontana, Krueger references
**[[journalling|EPSE 595]]
**workbook #2
!!29 February
*CPSC 313 9-10am DMP 110
*meeting w/ MS: [[HDD-DR Ethnographic Project]] - 10:30-11:30am (- TM, sick)
**updates to QA table: split nonlinear DR column, algorithmic input + implicit data analysis
**methodology token - slides describing the methodology
*[[EPSE 595]]
**lecture on interviewing
**workbook #2 due
!!01 March
*meeting w/ TM re: topics for Overview, ~MoDisco project discussion: [[minutes|TM-12.03.01]]
**to do: read final ~MoDisco submission, reviews, read Yang, OST (3D landscape) papers,
**to do: generate questions for JS re: tags, clusters, tasks, motivation
*Graphical Inference for ~InfoVis: research, who's cited, etc.
*CPSC 313
**test #3
**tutorial #6
**assignment #4
!!02 March
*CPSC 313 9-10am DMP 110
*meeting w/ MS for [[HDD-DR Ethnographic Project]]
*~InfoVis group meeting: SI pre-paper talk
!!References
<<bibliography>>
!!05 March
*[[Overview Discussion 12.03.05]] + prep
*lunch w/ JS, MS
!!06 March
*email catchup
*[[HDD-DR Ethnographic Project]]
**Fail/Struggle/Happy encoding on taxonomy, overlay
**methodology
*[[EPSE 595]]
**reviewing Krueger reference
**[[journalling|EPSE 595]]
*CPSC 313
**test #3
**tutorial #7
**assignment #4,5
!!07 March
*methodology token - slides describing the methodology
*[[EPSE 595]]
**lecture on group interviewing
**workbook #3 assigned
!!08 March
*meeting w/ TM + JM re: Overview debrief: minutes
*CPSC 313
**test #3
**tutorial #7
**assignment #4,5
!!09 March
*CPSC 313 9-10am DMP 110
*meeting w/ MS for [[HDD-DR Ethnographic Project]]
*[[EPSE 595]]
**workbook #3 - interviewing
!!References
<<bibliography>>
!!12 March
*CPSC 313 9-10am DMP 110
*[[HDD-DR Ethnographic Project]]
**meeting w/ TM, MS
*LUNCH meeting
*[[EPSE 595]]
**workbook #3 - interviewing, transcription
**[[journalling|EPSE 595]]: [[Material & Environmental Data]]
!!13 March
*[[EPSE 595]]
**workbook #3 - interviewing, transcription
**[[journalling|EPSE 595]]: [[Material & Environmental Data]]
*[[HDD-DR Ethnographic Project]]
**writing process, RW sections
!!14 March
*CPSC 313 9-10am DMP 110
*~C-TOC meeting (conflict)
*[[HDD-DR Ethnographic Project]]
**meeting w/ TM, MS
*[[EPSE 595]]
**lecture on [[Material & Environmental Data]]
**workbook #4 assigned
!!15 March
*CHI madness preparation
*meeting w/ JM re: CHI madness presentation
*CHI bookings
*[[HDD-DR Ethnographic Project]]
**writing process, RW sections
*reading DSM draft for ~InfoVis group meeting
!!16 March
*CPSC 313 9-10am DMP 110
*meeting w/ MS for [[HDD-DR Ethnographic Project]]
*~InfoVis group meeting: MS + MM + TM DSM paper draft
*[[HDD-DR Ethnographic Project]]
**writing and revising
!!References
<<bibliography>>
!!19 March
*CPSC 313 9-10am DMP 110
*no LUNCH meeting (cancelled, JM away)
*[[DRITW|HDD-DR Ethnographic Project]]
**meeting w/ TM, MS
**updates to figures (done)
**writing and revising pass
***add refs to empty citations (use Charmaz ref for GT)
***add numbers/counts/proportions to discussion section
**to do: supplementary material (not urgent)
***use SPLOM paper template (MS to send)
***replicate slides of methodology from pre-paper talk
***table of source material for each user scenario
!!20 March
*[[EPSE 595]]
**workbook #4 - found data / material culture
**[[journalling|EPSE 595]]: [[Organizing and Making Sense of Data]]
!!21 March
*CPSC 313 9-10am DMP 110
*[[EPSE 595]]
**lecture on [[Organizing and Making Sense of Data]]
*[[Overview]] - field / deployment study interview questions
!!22 March
*meeting w/ TM re: [[Overview]] project field / deployment study interview questions
*[[Overview]] - field / deployment study interview questions
*CHI madness final preparation
*reading [[DRITW|HDD-DR Ethnographic Project]] draft for ~InfoVis group meeting
!!23 March
*CPSC 313 9-10am DMP 110
*[[Overview]]
**1st potential user out of the woodwork - reporter from Tulsa, OK
**[[Leaksplorer|http://www.leaksplorer.org/beta]] in beta (U of T ML project - [[@leaksplorer|http://twitter.com/#!/leaksplorer]] - asked to join beta
*meeting w/ MS for [[DRITW|HDD-DR Ethnographic Project]] - cancelled
*~InfoVis group meeting: MS + MB + TM [[DRITW|HDD-DR Ethnographic Project]] paper draft
*[[DRITW|HDD-DR Ethnographic Project]] - supplemental material
*CS 313 studying
**tutorial #9,10
**test #3,4
***assignments #4,5,6
*[[DRITW|HDD-DR Ethnographic Project]]
**meeting w/ MS, TM 4pm
!!References
<<bibliography>>
!!26 March
*[[Overview]] - JS email
*no LUNCH meeting (cancelled, faculty recruiting)
*[[DRITW|HDD-DR Ethnographic Project]]
**updates to figures
**writing supplementary materials
*(feeling ill, home early)
!!27 March
*(still ill, at home)
*booking CHI flights
*[[EPSE 595]]
**[[journalling|EPSE 595]]: [[Computer Assisted Data Analysis, Data Displays]]
***downloading and trying out [[HyperRESEARCH|http://www.researchware.com/products/hyperresearch]]
***Tutorials 1,2,3,4
** [[workbook #5|EPSE 595 Workbook 5]] - data analysis
**//Research proposal//: [[Overview Deployment Field Survey]]
*[[Overview]]
**[[Overview Deployment Field Survey]] - emailed JS w/ follow-up
**[[LeakExplorer|http://www.leaksplorer.org/]]
**[[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
**[[Reporters' Lab|http://www.reporterslab.org/]]
**[[Sarah Cohen|http://fds.duke.edu/db/Sanford/faculty/sarah.cohen]] on Orientation - NICAR talk on text analytics
**[[A Story-based Approach to Making Sense of Documents|http://www.actapress.com/Abstract.aspx?paperId=451928]], DOI: [[10.2316/P.2011.747-013|http://dx.doi.org/10.2316/P.2011.747-013]] by Eric Bier, William Janssen, Patricia Wall, Jonas Karlsson, Tong Sun, Wei Peng, and Zahra Langford. In Proceeding (746) Internet and Multimedia Systems and Applications / 747: ~Human-Computer Interaction - 2011
*Reading SI's ~DistMatrix paper draft for ~InfoVis group meeting
!!28 March
*Reading SI's ~DistMatrix paper draft for ~InfoVis group meeting
*CPSC 313 9-10am DMP 110 - did not attend
*~InfoVis group meeting 10:30 - SI ~DistMatrix / Glint paper draft
*[[EPSE 595]]
**lecture on [[Computer Assisted Data Analysis, Data Displays]]
*[[DRITW|HDD-DR Ethnographic Project]] - writing token
!!29 March
*[[DRITW|HDD-DR Ethnographic Project]] - writing token
!!30 March
*CPSC 313 9-10am DMP 110
*[[DRITW|HDD-DR Ethnographic Project]] - writing token
*meeting w/ MS 10am
*meeting w/ TM + MS 11am
*~BibTex token, figures token
!!31 March
*[[DRITW|HDD-DR Ethnographic Project]] - submission deadline 5pm
!!References
<<bibliography>>
!!02 April
*CPSC 313 9-10am DMP 110
*[[Overview]] - JS email
*LUNCH meeting
*[[EPSE 595]]
**[[journalling|EPSE 595]]: [[Representing Knowledge]]
**[[workbook #5|EPSE 595 Workbook 5]] - data analysis
**//Research proposal//: [[Overview Deployment Field Survey]]
!!03 April
*[[EPSE 595]]
**[[journalling|EPSE 595]]: [[Representing Knowledge]]
**[[workbook #5|EPSE 595 Workbook 5]] - data analysis
*[[Overview]] / [[EPSE 595]]
**[[Overview Deployment Field Survey]]
**[[LeakExplorer|http://www.leaksplorer.org/]]
**[[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
**[[Reporters' Lab|http://www.reporterslab.org/]]
**[[Sarah Cohen|http://fds.duke.edu/db/Sanford/faculty/sarah.cohen]] on Orientation - NICAR talk on text analytics
**[[A Story-based Approach to Making Sense of Documents|http://www.actapress.com/Abstract.aspx?paperId=451928]], DOI: [[10.2316/P.2011.747-013|http://dx.doi.org/10.2316/P.2011.747-013]] by Eric Bier, William Janssen, Patricia Wall, Jonas Karlsson, Tong Sun, Wei Peng, and Zahra Langford. In Proceeding (746) Internet and Multimedia Systems and Applications / 747: ~Human-Computer Interaction - 2011
!!04 April
*CPSC 313 9-10am DMP 110
*[[EPSE 595]]
**lecture on [[Representing Knowledge]]
**reading <<cite Charmaz2006 bibliography:Bibliography>>
!!05 April
*meeting w/ JM: [[minutes|JM-12.04.05]]
*[[Overview]] / [[EPSE 595]]
**[[Overview Deployment Field Survey]]
**reading <<cite Charmaz2006 bibliography:Bibliography>>
!!06 April
*(Good Friday holiday)
!!References
<<bibliography>>
!!09 April
*(Easter Monday holiday)
!!10 April
*Access control problems - MUX, IMAGER access expired
*CW Anderson email
*LUNCH meeting
*JD on recording Skype: [[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]]
*TM sent:
**Keogh, E. [[How to do good research, get it published in SIGKDD and get it cited!|http://www.cs.ucr.edu/%7Eeamonn/Keogh_SIGKDD09_tutorial.pdf]] (slide deck)
**[[Bring Back the 40 Hour Work Week|http://www.salon.com/2012/03/14/bring_back_the_40_hour_work_week/singleton]] by Sara Robinson
*[[EPSE 595]]
*[[Overview]] / [[EPSE 595]] //Research proposal//
**[[Overview Deployment Field Survey]]
**Material from J Stray:
***[[LeakExplorer|http://www.leaksplorer.org/]]
***[[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
***[[Reporters' Lab|http://www.reporterslab.org/]]
***[[Sarah Cohen|http://fds.duke.edu/db/Sanford/faculty/sarah.cohen]] on Orientation - NICAR talk on text analytics
***[[A Story-based Approach to Making Sense of Documents|http://www.actapress.com/Abstract.aspx?paperId=451928]], DOI: [[10.2316/P.2011.747-013|http://dx.doi.org/10.2316/P.2011.747-013]] by Eric Bier, William Janssen, Patricia Wall, Jonas Karlsson, Tong Sun, Wei Peng, and Zahra Langford. In Proceeding (746) Internet and Multimedia Systems and Applications / 747: ~Human-Computer Interaction - 2011
**Material from CW Anderson:
***[[Truth, documents and data journalism鳴ory|http://www.reporterslab.org/cw-anderson/]], [[Reporters' Lab|http://www.reporterslab.org/]]
***[[The Things That Tell Us Whatⵥ (a Little Research Manifesto)|http://journalismschool.wordpress.com/2011/03/11/the-things-that-tell-us-what-is-true-a-little-research-manifesto/]]
***<<cite Anderson2013 bibliography:Bibliography>>: Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism.
***Additional reads:
****[[From Indymedia to Wikileaks: What a decade of hacking journalistic culture says about the future of news|http://www.niemanlab.org/2010/12/from-indymedia-to-wikileaks-what-a-decade-of-hacking-journalistic-culture-says-about-the-future-of-news/]] by CW Anderson, [[Nieman Journalism Lab|http://www.niemanlab.org/]]
****[[To build a digital future for news, developers must be able to hack at the core of old systems|http://www.niemanlab.org/2011/03/matt-waite-to-build-a-digital-future-for-news-developers-have-to-be-able-to-hack-at-the-core-of-the-old-ways/]] by M Waite, [[Nieman Journalism Lab|http://www.niemanlab.org/]]
****[[Audience Atomization Overcome: Why the Internet Weakens the Authority of the Press|http://archive.pressthink.org/2009/01/12/atomization.html]] by J. Rosen, [[PressThink|http://pressthink.org/]]
****[[The New Precision Journalism|http://www.unc.edu/~pmeyer/book/]] (book), [[Public Journalism and the Problem of Objectivity|http://www.unc.edu/~pmeyer/ire95pj.htm]] by P. Meyer
****NYT: [[A Selection From the Cache of Diplomatic Dispatches|http://www.nytimes.com/interactive/2010/11/28/world/20101128-cables-viewer.html?ref=wikileaks#report/cables-09KABUL3068]], [[All the Aggregation That鴠to Aggregate|http://www.nytimes.com/2011/03/13/magazine/mag-13lede-t.htm?_r=2]] by B. Keller, [[WikiLeaks|http://topics.nytimes.com/top/reference/timestopics/organizations/w/wikileaks/index.html?scp=1-spot&sq=wikileaks&st=cse]]
****Huffington Post: [[Why The New Republic is Wrong on Aggregation|http://www.huffingtonpost.com/robert-teitelman/the-new-republic-on-aggre_b_833105.html]] by R. Teitelman
**created [[References: Overview]]
!!11 April
*[[EPSE 595]] //Research proposal// due
*reading: <<cite Anderson2013>>: Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism.
*~C-TOC meeting 11am, Koerner pavilion of UBC hospital
*MUX meeting
*participating in SH's study
*reading <<cite Frankfurt2005>>
!!12 April
*reading [[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
*meeting w/ JM + TM: [[minutes|TM-JM-12.04.12]]
*reading paper for ~InfoVis group discussion: <<cite Sprague2012>>, <<cite Sprague2009>>
**adding to [[References: To Read]]
!!13 April
*~InfoVis group meeting: paper discussion: <<cite Sprague2012>>, <<cite Sprague2009>>
**emailed RR re: RPE
***response (agreed to supervise), and:
>BTW, you might be interested in knowing that VIVA has been developing some connections to the journalism world as well. For example, we have connections with the director of the Journalism school here at UBC (a former editor of the Globe and Mail, btw), and have been doing some projects with the Vancouver Sun. It might be good to compare notes at some point. (If nothing else, we could explore the possibility of funding some work to help you in this area...)
**emailed CWA re: collaboration
**emailed JS re: insight study, coordinating / piloting interview study
*meeting w/ CT re: interruptions journal paper 1pm
*reading <<cite Charmaz2006>>
!!References
<<bibliography>>
!!16 April
*response from JS re: CWA collaboration
>//This is a really awesome conversation or eavesdrop. Enjoying both your work. And the first Overview user who isn't me is grappling with it right now. Also, Dan Cohen (Mr. Digital Humanities) and friends are trying to use it on some historical texts. So lots going on.//
*response from RR: re RPE, UBC journalism school
>some VIVA people have been supervising students working with the Vancouver Sun. RR also in touch w/ ~Mary-Lynn Young at UBC Journalism school.
*CHI presentation preparation
!!17 April
*CHI presentation preparation
*LUNCH meeting
*reading <<cite Charmaz2006 bibliography:Bibliography>> - coding
!!18 April
*reading <<cite Charmaz2006>> - coding
*CHI presentation preparation
*MUX meeting, CPSC 543 final presentations, social
!!19 April
*meeting w/ TM: [[minutes|TM-12.04.19]] (email update sent instead)
*checking out [[Dedoose|http://www.dedoose.com/]]
*checking out [[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]]
*browsing the CHI 2012 advance program, finding interesting papers to read
*CHI presentation preparation
*outline for domestic interruption journal paper
*reading <<cite Charmaz2006>> - coding - see [[notes|Grounded Theory]]
!!20 April
*CHI presentation preparation
*created [[Data Journalism: Links and Literature]] literature review / links page
*website updates
*taxes
*reading <<cite North2011>> - see [[notes|Grounded Theory]]
!!21 April
*~InfoVis group party 3pm
!!References
<<bibliography>>
!!23 April
*CHI presentation preparation
*reading:
**<<cite North2011 bibliography:Bibliography>> - see [[notes|Information Visualization Evaluation: Qualitative Methods]], [[Insight-Based Evaluation]]
**<<cite Endert2012>>: [[CHI 2012 on semantic interaction|http://people.cs.vt.edu/aendert/Alex_Endert/Research_files/Endert_CHI2012_Semantic_Interaction.pdf]]: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite O'Brien2010>>: insight-based evaluation for Gremlin: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!24 April
*CHI presentation preparation
*LUNCH meeting - 12pm x530
**GRAND research practice talks x3
**DT's study on haptic reminders - demonstration
!!25 April
*CHI presentation preparation
*~C-TOC meeting - 11:00am x530
*IMAGER social - 12:30pm x836
*MUX meeting, CHI practice talk - 2pm x836
**DT's study on haptic reminders - during MUX talk
*~InfoVis group meeting - 3pm x530
**discussion lead: on <<cite Endert2012>>
!!26 April
*CHI presentation preparation
*meeting w/ JM: [[minutes|JM-12.04.26]] - CHI practice talk debrief
!!27 April
*CHI presentation preparation
*follow-up w/ [[VIVA|http://www.viva-viva.ca/index.html]] re: UBC journalism
*follow-up w/ JS re: Overview studies
>//In the meantime I've been revisiting the insight-based evaluation literature. The most recent publication* from the original insight authors suggests a shift away from attempting to quantify insights generated by participants, or relying on domain experts to rank or grade insights, and towards a researcher-led qualitative open-coding of insights into conceptual categories. These categories could be validated at a later stage of analysis by a domain expert; in our case this person would likely be a journalism school faculty member. The appeal of this approach is that it will likely require less involvement of a busy faculty member, offloading most of the analysis to us the researchers, and would allow for a more qualitative comparison between the document mining experience with and without Overview. //
>
>//Nevertheless, it would still be great to have this insight study become part of a course project at a journalism school, should faculty member(s) be interested; if our research participants are receiving course credit for using Overview, we will have an easier time keeping them engaged and motivated longitudinally.//
!!References
<<bibliography>>
!!30 April
*writing [[An Insight-Based Evaluation of Overview]]
*CHI presentation preparation
*reading:
**<<cite vanderMaaten2011 bibliography:Bibliography>> for ~InfoVis group meeting
**<<cite Shipman1999>> - formality considered harmful;
*replay from JS: introduced to [[Dan Cohen|http://www.dancohen.org/]], Associate Professor in the Department of History and Art History at George Mason University and the Director of the Roy Rosenzweig Center for History and New Media.
!!01 May
*CHI presentation preparation
*response from JS re: User #0 - in the midst of analysis; conducting interviews; writing stories:
>//[JW] seems to be finally in the thick of the analysis, having just really gotten Overview working on his documents (which is itself a story, I think!) In any case, you can hopefully get a flavor of where we are from the below.//
>
>[...] //Looks like he's just really getting started. Getting the data set up for analysis -- sometimes called ETL (extraction, transformation, loading) -- is recognized to be a big deal in the commercial analytics world. It's something Overview has to deal with seriously as a project but I'm not quite sure how your study will or won't address this.//
*created [[Graphical Inference User Studies]] project
**[[compiled list of papers|Graphical Inference Evaluation]] citing original graphical inference paper, 2 papers conducting a graphical inference user evaluation:
***Wood et al (2011) - ~BallotMaps (TVCG)
***Kairam, Heer et al (2012) - ~GraphPrism (AVI)
**configuring [[Ggplot2|http://had.co.nz/ggplot2/]] for R, [[nullabor|https://github.com/ggobi/nullabor]] package for graphical inference
*no LUNCH meeting (GRAND AGM)
*response from JS re: interviews
**happy to do pilot interview
**''to do'': see AP Caracas bureau chief notes
**[[ITP thesis archivers using Overview|http://blog.itparchive.com/post/20561066325/experimenting-with-the-associated-press-developed]]
*reading:
**<<cite Furniss2011>> - confessions of a [[Grounded Theory]] ~PhD (CHI)
!!02 May
*CHI presentation preparation
*no MUX meeting (GRAND AGM)
*~InfoVis group meeting - 3pm x530: discussing <<cite vanderMaaten2011>>
!!03 May
*CHI presentation preparation
*meeting w/ TM: [[minutes|TM-12.05.03]]
*emailed JS re: [[Overview]] studies
!!04 May
*CHI presentation preparation
*CHI departure prep
*adding / exploring / reading [[Data Journalism: Links and Literature]]
*email from Fred Gibbs / Dan Cohen (digital humanities)
**attempted to use Overview to explore database of stories, peoples' response to 9/11, a 30MB CSV file (stuck, currently, haven't yet got data fully loaded, will try again)
**(seeking emerging patterns over space (region) and time (Overview does not currently allow exploration of multiple time slices, multiple geographical points)
***possible compromise to tag data with geospatial and temporal tags
**also used Overview to explore data from Google Books: dictionaries, medical texts (huge dataset)
***again seeking ability to track temporal change
!!References
<<bibliography>>
!!07 May - 10 May
*At CHI, Austin TX
!!11 May
*(in Dallas, TX, visiting relatives)
!!References
<<bibliography>>
!!14 May
*(in Dallas, TX, visiting relatives, returning to Vancouver)
*reading <<cite Krzywinski2011 bibliography:Bibliography>> for ~InfoVis group meeting
!!15 May
*CHI proceedings mining / finding papers to read
*CHI expenses sorting
*LUNCH meeting 12pm in x60
*UBC awards annual progress report
*paid UBC fees Spring/Summer 2012
*[[Overview]] project
**catching up on email w/ JS
**responding to JS, arranging pilot interview
>there would be two study designs, however they would be similar in many respects. Some key differences:
>
>"in the wild" users ala Jarrel Wade would use Overview with their own data, while we would provide the dataset for students to work with.
>students would get both study conditions, Overview and search-only. We will rely on retrospective anecdotes from "in the wild" Overview users about prior search-only projects (if any).
>students' time would need to be constrained in order to align with their academic calendar, while we don't have as much control over how long "in the wild" users spend with Overview, as many may be working with deadlines
>
>The two studies would be similar in terms of data collection and analysis methods. In both cases we will conduct interviews with users and collect artifacts (log files, screen captures), and the analysis of this data will be similar for both studies, facilitating comparisons between students and "in-the-wild" Overview users.
>
>@@color:#444bbb; ''JS'': Why do you want to have each subject try both methods?@@
>
>A few reasons. First, it's a tradeoff: a within-subjects study would require less participants but more time per participant. It will allow us to ask all participants to make comparisons between both methods; we'll be able to study whether the ordering of methods has an effect on the resulting analysis. It will also allow us to better detect particularly strong and weak students (regardless of method).
>
>Due to the longitudinal aspect of this study, we won't be able to use a think-aloud approach. Instead, we'll rely on a guided retrospective walkthrough of their analysis via video chat. Re: keeping notes and understanding what constitutes insight, we can provide participants with some initial prompts or characteristics of insight. A few of the published insight studies have asked participants to list potential findings, hunches, or questions about the data before exploring a dataset; whenever an item on this list is addressed during their analysis, they would be encouraged to take note. Maintaining regular notes could also be a requirement of the course assignment.
*revising <<cite Krzywinski2011 bibliography:Bibliography>> for ~InfoVis group meeting
!!16 May
*created / annotated [[References: CHI2012]]
*MUX meeting: Mary Muir
*~InfoVis group meeting - 3pm x530: discussing <<cite Krzywinski2011>>
*Torsten M⠨SFU gruvi lab) practice talk (~EuroVis capstone address)
!!17 May
*preparing talk for IMAGER social
*meeting w/ TM: [[minutes|TM-12.05.17]]
*CHI expense finalizing
*website updates / IMAGER pubs updates
!!18 May
*using Overview, examining log files
*preparing talk for IMAGER social
*UBC awards annual progress report
*revising [[References: CHI2012]] - sent out to ~InfoVis group
!!References
<<bibliography>>
!!21 May
*(Victoria Day holiday)
*reading <<cite Eisenstein2012 bibliography:Bibliography>> (2012 CHI ~TopicViz WIP): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
!!22 May
*notes for <<cite Eisenstein2012>>
*T. M⠲e: Anoop Sarkar @ SFU:
**TMailable: Jun 3-15; AS unavailable May 27 - Jun 7; Jun 20 - Jul 21 (window: Jun 16-19; May 23-25)
>@@color:#444bbb; ''AS'': //"Overview is really interesting: the best thing I like about it is that you have a subject expert who is integrated into your design process. I would like to discuss things from a different perspective, that of natural language processing of the underlying text, if you are interested."//@@
**this makes me curious as to who the intended end users are, what are their tasks, what else is involved in their research / analysis process. There may be an evaluation project lurking somewhere in here. I'll keep this in mind.
>@@color:#444bbb; ''AS'': //"it is an early prototype and we are rapidly developing and testing new design decisions. [...] My purpose is to make discovery of useful tuples such as <(person/place/thing) , (relationship/event) , (to person/place/thing), time, location> from large document collections. We ignore other useful characteristics of events currently."//@@
*UBC awards annual progress report (adding TM/JM's contribution)
>"The first term of my ~PhD involved an extensive literature review of evaluation methodologies and techniques applicable to visualization tools. Also during this time I began collaborating with a post-doctoral fellow, assisting him in the analysis of data from a series of interviews he conducted with users of dimensionality reduction techniques. This resulted in a co-authored paper [1], which was recently submitted to the IEEE ~InfoVis conference (Fall 2012). Meanwhile, I began preparing for an observational field study of fisheries scientists, an evaluation of a new fishery simulation visualization application; unfortunately, this project has been postponed due to logistical issues. In my second term, I completed an elective graduate course in qualitative research methods. My final project for this course was a research proposal for a study that will begin this summer: a series of interviews with users of a visualization tool for text document mining. I am also planning a second, parallel study: a longitudinal evaluation of this application with journalism students. In Fall 2012, I will present both studies to my supervisory committee, satisfying my department's research proficiency evaluation for 1st year ~PhD students. In May 2012, I presented a paper emanating from my M.Sc research [2] at the ACM SIGCHI conference. In the 2012-13 winter term, I intend to take a graduate course in operating systems. After which, the course requirement for my degree will be satisfied. To prepare for this course, I attended a term of lectures for an undergraduate course in operating systems. [1] Michael Sedlmair, Matthew Brehmer, Stephen Ingram, and Tamara Munzner. Dimensionality Reduction in the Wild: Gaps and Guidance. Submitted to ~InfoVis 2012. [2] http://goo.gl/427yJ"
>@@color:#444bbb; ''JM (+TM)'': //"Matthew's report provides an accurate and detailed account of his progress, and so there is no need to reiterate those details in this section. Prof Tamara Munzner and I are co-supervising Matthew and have no concerns about his progress in the ~PhD program.//@@
*preparing talk for IMAGER social
*LUNCH meeting 12pm (cancelled - Friday instead)
*CSGSA annual general meeting 1pm
*[[Overview]] project software:
**[[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]] (~20$)
**[[dedoose|http://www.dedoose.com/]] - JW, CT, MH recommended (web app, $13 monthly subscription service, 15% discount if bought for 6mo/12mo), has transcription functionality: [[sign-in here|https://www.dedoose.com/App/?Version=4.2.82]]
***[[user guide|https://www.dedoose.com/Support/UserGuideHTML.aspx]]
**[[Skype Premium|http://www.skype.com/intl/en-us/prices/premium?intcmp=CS-Upsell-NarrowRight]] ($60 / annually)
*familiarizing myself with [[dedoose|http://www.dedoose.com/]] (adding documents, descriptors, linking documents with descriptors)
!!23 May
*~C-TOC meeting 11am x530
*IMAGER social - MB WIP talk
*MUX meeting: MH practice talk for GI
*~InfoVis group meeting - 3pm x530: MS practice talk for ~EuroVis (SPLOM)
*Overview for music discovery (fun little side project)
**[[scraperwiki.com|https://scraperwiki.com/profiles/mattbrehmer/]] - PHP / python web scraper
**mining the Rolling Stone Top 500 Albums of all time in Overview
**mining Pitchfork Album reviews in Overview
!!24 May
*meeting w/ JM: minutes
*Overview for music discovery (fun little side project)
*browsing the [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]
*contacted VA challenge group leader:
>"SN reminded me yesterday about some of your group's projects, so I've been reading your VA challenge site. I'm particularly interested in the DTES and political text analysis projects - are these projects still ongoing? Ron had mentioned that some students were collaborating with the Vancouver Sun. I'd be curious to read an any news articles that have resulted from this collaboration, and to find out how VA tools factored in to the process.
>
>I'm currently working with a collaborator at the Associated Press who has been developing Overview (http://overview.ap.org), a tool for exploratory data analysis and visualization of large text document collections. It's freely available through the site, and we're looking for users. I'm conducting a longitudinal evaluation studies of the tool.
>
>Let me know if yourself, anyone in the VA challenge group, or anyone at the Vancouver Sun is interested in using Overview in your projects.
*[[How the AP's Overview Turns Documents Into Pictures|http://www.pbs.org/idealab/2012/05/how-the-aps-overview-turns-documents-into-pictures144.html]] - PBS Media Shift / Knight Foundation Idea Lab
*[[http://www.unz.org/]] - archived historical PDF periodicals ([[CWA via twitter|https://twitter.com/#!/Chanders]])
*installed [[homebrew|https://github.com/mxcl/homebrew/wiki/installation]], [[docsplit|http://documentcloud.github.com/docsplit/]] (a document cloud utility), dependencies: ~GraphicsMagick, Poppler, Tesseract (for PDF to CSV conversion for Overview); ~XCode 4.3 + command line tools, Java Developer Update
**[[docloader|https://github.com/overview/overview-prototype/blob/master/docloader/docloader.rb]]
*emailed Jonathan re: logging, pilot interview:
>- select document in MDS view (allowing us to compare how often this is done vs. selecting a document in the document list)
>- select node in node folder list (allowing us to compare how often is this done vs. selecting a node in the tree view vs. selecting documents in the MDS view)
>- select tag in tags view list
>- save tag file (allowing us to gauge how often users save their tags, complementing the 'load tag file' logging)
>- select random button (which will help us to answer how often this feature is used vs. selecting a node by hand from the document list panel)
>- cluster button (I'd be curious to know how often users re-cluster)
!!25 May
*wrote batch txt to PDF script; using Overview to visualize ~2000 news articles from various media outlets relating to Vancover's DTES (from VA challenge site)
*Overview for music discovery
*LUNCH meeting
!!References
<<bibliography>>
!!28 May
*Downloading / exploring Jeff Heer's C3 colour naming tool package
*Overview: DTES analysis
*Overview: music review exploration discovery / [[blog post writing|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]]:
>Do I have to read or skim thousands of music reviews to get a sense of 䠩s contemporary music?诠are the big names and where do they come from? What are the trends? How do genres intersect?
>
>Could these exploratory questions be answered visually?
>
>Each point in the scatterplot above represents the body text of a record review scraped from [[P4K|http://pitchfork.com]] (1999 Ჩ. The placement of each point reflects the corresponding review魩larity with other reviews.
>
>I祮erated this visualisation using [[Overview|http://overview.ap.org]], a free tool for searching and exploring large collections of text documents, developed by Jonathan Stray at the Associated Press, in collaboration with Stephen Ingram and our mutual supervisor Tamara Munzner at UBC. A recent technical report explains some of its core algorithms in much greater detail than what I offer here.
>
>Each document, a record review in this case, is converted into a list of words and word-pairs appearing in that document. Common English words are discarded, such as articles, conjunctions, and pronouns. It襮 possible to compute the 䡮ce�one document to another, based on the similarity of the two documentﲤ lists. As a result you have a massive high-dimensional space of inter-document distances.
>
>In order to show all documents on a screen in 2 dimensions, we can use one of a number of techniques for transforming a high-dimensional space to a 2-dimensional space, while preserving inter-document distances.
>
>In the resulting scatterplot, the absolute position of documents has no meaning: there are no up-down or left-right axes. Only the relative distance between points is important. This means that as someone looking at a plot like this, your goal would be to look for distinct cluster structure, rather than correlation, as you might do in your typical standard 2-axis scatterplot.
>
>That I got an enormous blob is hardly surprising: the language used in record reviews tends to be pretty similar. Maybe if I started with 5,000 record reviews and 5,000 movie reviews, I might have seen 2 distinct clusters. But an ambiguous blob at the macro-level doesn壥ssarily mean that theretructure at the micro-level.
>
>A quick pass of tagging and colouring by genre was performed manually. The cluster of red are ~20 releases recorded or produced by Kieren Hebden (aka Four Tet). The little red cluster of Four Tet records is a case in point. While the larger genres such as electronic or hip-hop are spread all over the plot, Iᬳo found similarly tiny clusters relating to niche genres, such as sludge metal or ~ATL-area hip hop.
>
>So how did I find these micro-level clusters? What you can嬬 from the screen shot above is that the application has panning and zooming controls, and every time I click on a point I pull up the corresponding full-text review in a separate window. There쳯 a second visualisation in this application which displays cluster structure in a tree-like hierarchy, which makes these smaller clusters easier to spot.
>
>I agree with some of the comments posted here: scatterplots may not be the best choice (some of the time). It really depends on what your task is. If you쯯king for correlation, a density plot may be a better idea. Other times, a scatterplot might only be used to get a preliminary overview, used in conjunction with other visual analysis techniques.
>
>My current research project involves studying individuals who have a need to explore large text document collections. I髥 to hear from journalists, digital humanities researchers, archivists, and librarians who (a) have this need and (b) give Overview a try. This will help us to build upon and improve Overview, as well as future tools for addressing these types of tasks.
*reading:
**<<cite Jianu2012 bibliography:Bibliography>> - small UI changes affect analytic strategies (CHI 2012): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**[[Don't use scatterplots|http://www.chrisstucchio.com/blog/2012/dont_use_scatterplots.html]] by Chris Stucchio + [[discussion board comments|http://news.ycombinator.com/item?id=4027201]]
**[[Anyone can do it. Data journalism is the new punk|http://www.guardian.co.uk/news/datablog/2012/may/24/data-journalism-punk#_]]
***tools mentioned: [[Outwit Hub|http://www.outwit.com/]] (scraping tool); [[Gephi|http://gephi.org/]], [[Google Refine|http://code.google.com/p/google-refine/]]
*Experimenting w/ [[Google Fusion Tables|https://www.google.com/fusiontables/DataSource?docid=1iU_t_bmnXu5rdIP5ceN0U8nP3LF_LEk9cFj1NQw]]
**<<cite Willett2012>> - strategies for crowd sourcing social data analysis (CHI 2012): [notes|Information Visualization Evaluation: Quantitative Methods]]
!!29 May
*Further learning [[Google Fusion Tables|https://www.google.com/fusiontables/DataSource?docid=1iU_t_bmnXu5rdIP5ceN0U8nP3LF_LEk9cFj1NQw]]
*Overview w/ Vancouver city council meeting minutes (from VA challenge group web site)
*LUNCH meeting 12pm
*Tuesday Tea
*reading <<cite Brandes2012>> (Gestaltlines) for ~InfoVis
!!30 May
*Overview w/ Vancouver city council meeting minutes (from VA challenge group web site)
*notes for <<cite Jianu2012>>, <<cite Willett2012>>, <<cite Brandes2012>>
*browsing the [[Canadian Communication Association|http://www.acc-cca.ca/Default.aspx?pageId=1309770]] 2012 conference program via @mckelveyf (will chat with next week)
*~InfoVis group meeting - 3pm x530: discussing <<cite Brandes2012>> [[Gestaltlines|Information Visualization: Techniques]]
*reply from JS:
>@@color:#444bbb; ''JS'': //"Jarrel's story is to run Sunday. Will include a sidebar describing methodology. I think he also plans a blog post somewhere. When do you want to get things rolling? Should we do a skype chat this week and then schedule Jarrel for next week?.//@@
*my response:
>That's great to hear. I'm able to Skype tomorrow before 1 PDT or Friday afternoon between 12:30 and 5 PDT. I'll be fine with interviewing Jarrel any day next week.
>
>Regarding the music reviews dataset, most of the effort was spent learning how to use ~ScraperWiki. The Overview UI was less responsive with a dataset of this size, but it worked nevertheless. I've also been exploring the dataset in Google Fusion Tables: http://goo.gl/ISOf4 - the CSV file is downloadable from there.
>
>I've also been using Overview to explore a couple of smaller datasets posted on the website of UBC undergrad visual analytics group: (1) meeting minutes from ~130 Vancouver city council meetings from 2005-2011, split into 2,554 pages, with max 2-3 meeting topics discussed on each page. (2) ~1,400 articles from Canadian news outlets relating to Vancouver's troubled downtown east side. There's some distinct structure in both datasets. I'm waiting to hear back from students in the VA group, as it appears as though they've been collaborating with journalists at the Vancouver Sun to analyze these datasets.
>
>My experience so far has me thinking about how my open-ended "let's see what's in this dataset" exploration might differ from journalists with hunches or stories in mind a priori. It also had me thinking comparatively about qualitative data analysis software used in humanities research, about the functionality they provide for open-ended / exploratory analysis. We could chat about these ideas as well.
!!31 May
*meeting w/ TM+JS 11am via Skype (pilot interview)
**tech issues: G+ hangout used instead w/ 3rd party screen recording
*meeting w/ CT 2pm (re: interruptions journal paper)
*reply from Kyle Melnick (VIVA challenge group):
>@@color:#444bbb; ''KM'': Thanks for the email. The project with the Vancouver Sun ended up being more of a visualization project than an analysis project (where it would be used for investigative analysis). We created a cluster-time analysis visualization that they posted around election time. That's not to say that they weren't interested in using VA tools for investigative analysis, in fact they were extremely interested, but it was more of a time commitment issue. So no one is currently working on a project with them but it wouldn't be hard to get something started, if the commitment was there.@@
>
>@@color:#444bbb; There was a report done for the Carnegie Center on media attitudes towards the downtown eastside, but that is more research based as opposed to supporting story generation in a journalism context. I can track down the latest version and double check whether I can send it to you.@@
>
>@@color:#444bbb; I took a quick look at Overview (cool stuff!) and I do think it would be useful for some of the text analytic projects we are currently working on. What might work best is maybe you can give a demo of the tool during one of our Challenge meetings, so that the students know the analytic possibilities/general functions of the tool. We also work with Anoop Sarkar, who specializes in natural language processing at SFU and is developing some interesting text analytics tools. I think it would be worthwhile meeting with him or one of his students. Check out his lab website http://natlang.cs.sfu.ca/ @@
>
>@@color:#444bbb; As you're interested in the visual analysis of data, the Vancouver Institute for Visual Analytics is hosting a brief summer course at the end of july for interested students. It will involve talks and workshops on visual analytics and related fields. If you're interested I can send you the registration info.@@
>
>@@color:#444bbb; I'm on campus mondays/tuesdays if you wanted to meet to talk more about us working with Overview and some other collaborative possibilities.@@
!!01 June
*reply to Kyle Melnick (VIVA challenge group):
>The Vancouver Sun visualization you mentioned is no longer on the news article page, can I find it anywhere else? Please keep me in the loop in the future if your group hears from anyone at the Vancouver Sun, or any other news outlet, using or requiring VA tools during their investigation.
>
>It would be great to see the Carnegie report, should it be accessible to me. Is any of this up on their site?
>
>I'd be happy to give a demo of Overview to the Challenge group at one of your meetings. This past week I've been using it to explore a couple of datasets on your site: the DTES news articles and the Vancouver city council meeting minutes. I'd be interested if you find some uses for it, and I'd like to have critical discussions of its utility, usability, and fit into VA workflows. Also, I'll be conducting a study using Overview with journalism students in the fall, so it could be a good opportunity to introduce the tool to another pilot audience before then. When does your group meet?
>
>Coincidentally, Tamara and I have just recently been put in touch with Anoop via Torsten M⠦rom SFU's gruvi lab. Anoop has expressed interest in Overview as well, as there may be some overlap with his Lensing Wikipedia project. We expect to talk to him in more depth once he's back from his current travels.
>
>Regarding the workshop - who is the intended audience? Ugrad students in CS / Cogs?
*Skype support boards: why does group screen sharing drop audio quality? Apparently an issue introduced in Skype 5.7+
*Saul Greenberg visit / demo / talk 11am: //proxemic interactions - the new ubiquitous computing?//
*meeting w/ JM + CT re: interruptions journal paper
*met Keith Rozendal, Vancouver Sun reporter covering Saul's visit
*chat w/ Saul in x508 2:45 - 3:00
*UDLS
!!References
<<bibliography>>
!!04 June
*Overview - JW
**reading [[JW's story|http://tulsaworld.com/tpdtechemails]], [[short version|http://www.tulsaworld.com/news/article.aspx?subjectid=298&articleid=20120602_298_0_Aftert169838]]
***49 reader comments, highlights:
>''Tought but fair'': //Finally, a very big Thank You to the Tulsa World, for finally opening up this long-whispered about and horribly expensive TPD fiasco to We the People - and for finally shining some truth and sunlight onto this costly problem. GO, Jarrell! TW readers are behind you 100%!//
>
>[later reverses opinion: ...] //I have taken considerable time to review each one of the e-mails made available by the Tulsa World and I have also re-read the TW article several times very carefully.//]
>
>[...] //It was interesting to read all of the e-mails provided by the Tulsa World. It appears that the ones made available to readers were hand-selected to illustrate the newspaper story's main points. That they were not presented sequentially in date order, however, made it difficult to develop any independent sense of whether some of the implications suggested by the TW story had any real basis in fact. As the level of technology background/knowledge of this TW reporter is also unknown, one did wonder whether perhaps important, objective, factual details relevant to the overall progress/success of this TPD project were, perhaps, somehow overlooked/omitted. I certainly found no evidence of anything other than a strong team of dedicated people, all working for our city, striving diligently to implement a complicated technology project, with a number of variables which were completely out of their power to correct or to control.//
>
>//The TW news story reports: 崨ics complaint about the Panasonic and Sprint 岴isement䠴o an internal investigation.頥mphasis on 岴isement>
>//However, after carefully reading each one of these published e-mails, I find that it was made specifically clear to Corporal Dalsing by Panasonic, from the very beginning, that if he agreed, and was approved by TPD, to share his project experiences in a proposed Panasonic case study about TPD岣hase and implementation of these Panasonic products, that both he and his TPD superiors would personally sign off on every step of the case study, and it was also explicitly stated to him that the 峠of the case study will be on positioning Tulsa PD as a forward-thinking agency using cutting-edge technology to address law enforcement challenges; NOT as an advertisement or product endorsement by the agency.奠Tulsa World e-mail extracts #0873.) //
>
>[...] //Or, perhaps this is simply a misunderstanding on the part of the Tulsa World reporter about the very significant difference which exists between an individual purchaser/user of some mainstream market product appearing in an 岴isement⠴hat commercial product - and an experienced, knowledgeable, technology team member, who is assigned to work on his employer駨ly complex deployment of a commercial vendor峳-than-robust technology products, also participating in a professional case study which details his/his organizationॣific hands-on experience with that vendorॣialized products or services 㡳e study which is designed to provide, via real-life examples, much more detailed information about the vendorॣific products or services. There is a very big difference between these two scenarios big 飨 an average layperson with little experience with modern, cutting-edge, technology simply might not be able to discern.//
>
>//I could make no logical connection whatsoever between Cpl DalsingᲴicipation, along with a TPD and City of Tulsa approved technology vendor, in customary knowledge-sharing activities which are routinely conducted throughout the entire technology industry, and the subsequent failure of that approved vendorĭpurchased products to be successfully deployed in the TPD patrol car project. And I certainly could find no supporting evidence in any of these published e-mails which would raise my own suspicions that Cpl Dalsing ever did anything at all unethical during the course of his difficult assignment to this complex, ambitious, TPD technology project. //
>
>''Cee2'': //Keep up the investigative reporting on this problem.//
***also, the [[project on DocumentCloud|http://www.documentcloud.org/public/search/projectid:%205269-tpd_emails]] (125 document subset + comments)
***[[commented source documents in Document Cloud|https://www.documentcloud.org/documents/330428-2010-2011-tpd-u1-comments-redacted.html#annotation/a51064]] (police discussion board)
**ran [[Overcloud|https://github.com/mtdukes/overcloud]] on 125 document subset in JW's [[project on DocumentCloud|http://www.documentcloud.org/public/search/projectid:%205269-tpd_emails]], imported into Overview
**JW available to meet today/tomorrow, will send notes Tuesday / Wednesday
**JW sent overview_log, tag file, source data csv, loading into Overview
***loaded document set CSV file (~6000 documents) into Overview, loaded tag file, examined log file
**re: screen sharing in G+ hangout:
>@@color:#444bbb; ''JW'': //It's a bit of a problem... one solution for me is to get our IT guys to let me through the firewall to download google video and borrow our "video station" webcam and mic. The other solution is just to do a video interview at our "video station" without any desktops sharing.//@@
>
>@@color:#444bbb; //What do you think, Matt? I'm happy to get working on it (the hardest part will actually be getting through the firewall -- the IT staff always make me jump through hoops.) Or would just doing a video chat get us where we need to go?//@@
*response:
>''MB'': It's worth a shot trying to get get permissions from your IT guys. While you're our first user, I know from past user experience evaluation projects that being able to see your screen will really help to communicate your thoughts on Overview's features and on your analysis process. The aim is to replicate, as much as possible, a meeting between the 3 of us at your office. With your permission, I'll be recording the video and audio from our session and using it to compare against future Overview users that we talk to.
>
>A possible alternative would be for me to provide you with a short list of questions pertaining to Overview's features and your analysis process. You could then record your answers in a short screencast that you narrate, or in a series of screenshots in a slide deck. While a recorded screencast or a series of screenshots wouldn't elicit spontaneous discussion, it's a compromise should you be unable to share your screen in real time. The downside to this option is the added time overhead, and I'd understand if you don't have the time for this.
>
>If you can't swing either option, we can still discuss without your screen share. Some of the other items I'd like to discuss wouldn't involve screen sharing at all: comparing Overview to other tools and processes, comparing your process for this story to those of past stories, and discussing your work context and background.
>
>Let me know which option works best for you. In the meantime, I'll be exploring your document set in Overview myself, going through your log file, and reading your notes when you send them.
>
>Thanks again for helping us out with this. Talk to you soon,
*JS blog post: [[How Overview turns Documents into Pictures|http://overview.ap.org/blog/2012/06/how-overview-turns-documents-into-pictures/]]
*email w/ Keith Rozendal, Vancouver Sun reporter / UBC journalism student - follow-up: Overview and the Vancouver City Council meeting minutes
>@@color:#324F17; ''KR'': //I plan to write Kyle or Scott when I feel like I have some time to play around with Overview. A presentation in the fall might be just the opportunity, so I'll push for sending you an invite to come over for a demo.//@@
>
>@@color:#324F17; //I think seeing your pre-processing strategies and the tools you used would be a nice element of a demo. At the moment, there's not much I could do with either the recipe or the final dish of docs, but maybe we can sit down later in the summer for a meeting where I can take a look. What sort of blocks on your time do you have in the next few weeks?//@@
>
>@@color:#324F17; //Given the local and political nature of the content, the minutes would be the perfect raw material for showing some of us at the j-school a bit about visual analytics.//@@
*sorting out CSGSA / UDLS permissions
*switched ~InfoVis slots w/ JD
**to read: <<cite Heer2012 bibliography:Bibliography>> - //Color Naming Models for Color Selection, Image Editing, and Palette Design// for ~InfoVis
*piloting G+ hangout + screen recording
*CT sent GI reviews for interruptions journal paper
!!05 June
*posted [[IMAGER social slide deck|http://cs.ubc.ca/~brehmer/proj/Imager_05.23.12.pdf]] with updates
*Overview/JW: continuing examining documents
**setting up a meeting w/ JW, JS - received notes - tentative 11am Wednesday
*LUNCH meeting 12pm
**LUNCH scrum [[blog|http://joannalunch.blogspot.ca/]] / [[tumblr|http://ubc-cs-lunch.tumblr.com/]]
*reading: <<cite Heer2012 bibliography:Bibliography>> - //Color Naming Models for Color Selection, Image Editing, and Palette Design// for ~InfoVis
*reading CT's GI reviews
*Tuesday Tea
!!06 June
*Overview meeting w/ JW, JS 11am
*MUX meeting
**OS's WIP talk: //Dr. Strangelog - or how I learned to stop worrying and love the IMU// - lessons learned from logging sensor data on Android devices
*~InfoVis group meeting - 3pm x530: discussing <<cite Heer2012>> - //Color Naming Models for Color Selection, Image Editing, and Palette Design//
*chatted w/ SI re: Overview tree view crowding / node placement
*meeting w/ JM+CT 4pm (Interruptions journal paper)
*DRITW rejected from ~InfoVis - fast tracked to TVCG
*chatting w/ @mckelveyf 7pm - communications ~PhD candidate at Ryerson
**http://www.measurementlab.net/visualization
**http://broadbandtest.eett.gr/
**http://www.onthemedia.org/blogs/on-the-media/2012/may/31/propublicas-message-machine/
**http://www.amazon.com/What-Computers-Cant-Artificial-Intelligence/dp/0060906138
**http://www.slate.com/articles/news_and_politics/victory_lab/2012/05/obama_campaign_ads_how_the_analyst_institute_is_helping_him_hone_his_message_.html
**http://www.onthemedia.org/blogs/on-the-media/2012/may/31/propublicas-message-machine/
**https://www.recordedfuture.com/this-is-recorded-future/how-recorded-future-works/
**http://mitpress.mit.edu/catalog/item/?ttype=2&tid=9945
**http://sonic.northwestern.edu/
**http://www.bivio.net/news/press_releases/2012/03062012-netfalcon.php
!!07 June
*meeting w/ TM+JM 11am: [[minutes|TM-12.06.07]]
*transcription preprocessing: fixing audio issues with interview
**strip audio from video using .mov to .aiff utility
**audio processing in Audacity: compression, selective amplification, noise removal, normalization, high-pass filter
**export back to aiff, recombine with video in iMovie, export as .mov
**convert .mov to .mp4 using Miro Video Converter
**upload to .mp4 to Depose
*OS's study 3pm
!!08 June
*JW interview transcribing
**Dedoose transcription functionality limited
**[[HyperTRANSCRIBE|http://www.researchware.com/products/hypertranscribe.html]] shop unsecured / paypal link not working
**downloaded veal license of [[inqscribe|http://www.inqscribe.com/]], requested student license (39$)
*UDLS
**presenting on [[dystopian fiction / literature|http://prezi.com/ccosrcyvorqw/this-will-not-end-well-dystopian-fiction/]] - learned how to use [[Prezi|http://prezi.com/index/]] presentation software
!!References
<<bibliography>>
!!11 June
*Overview / JW interview transcription
*activated [[InqScibe|http://www.inqscribe.com/]] student license
!!12 June
*expense submission for software / equipment purchases
*Overview / JW interview transcription
*LUNCH group meeting 3pm: brainstorming domestic interruption design implications
*Tuesday Tea 4pm (missed, LUNCH ran late)
*email from [[Prof. Alistair Sutcliffe, U. Manchester, UK|http://www.manchester.ac.uk/research/mbs/Alistair.sutcliffe/]] re: ~C-TOC and ongoing email monitoring
!!13 June
*responding to Prof. Sutcliffe
*review request from SIGGRAPH Asia
*~C-TOC meeting postponed until next week - 9am Jun 20
*no MUX meeting
*meeting w/ CT 2pm re: domestic interruption journal paper
*~InfoVis meeting 3pm
**JD on library sciences course
**~InfoVis group web presence
!!14 June
*Overview JW interview data analysis
*meeting w/ TM+JM: [[minutes|TM-JM-12.06.14]]
*emailing KR (Vancouver Sun / UBC jSchool)
*emailing KM (VIVA VA Challenge) - to demo Overview Monday 4pm Kenny 3308
*emailing JS re: debrief + next steps
*to do: ethics inquiry re: remote interviews
**http://research.ubc.ca/ore/breb-forms-guidance-notes
**contact: Shirley Thompson, Manager of BREB
**[[TCPS2 ch.10|http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/chapter10-chapitre10/#toc10-intro]]
**[[Guidance Notes on BREB Application|http://research.ubc.ca/sites/research.ubc.ca/files/uploads/documents/Ethics/BREB/Revised%20BREB%20guidance%20notes.pdf]]:
>''7.3.5 Expert interviews'', p. 28:
>
>Expert interviews are defined here as those that involve an interview with an expert in a similar position to the researcher (e.g., an academic, politician, owner or executive of a company, head of an NGO, or president of an association or union) and which are designed to obtain factual accounts of an event, a procedure, a process, history, and so forth, where there is minimal or no risk to the interviewee. If the person is being interviewed as an authorized individual who is qualified/able to release information or data about their organization and its policies, the research does not require review and the person does not need to complete a consent form, although professional interview procedures should be observed (see 䩥s exempt from review㴩on of the guidance notes). If the expert is being asked to proffer a personal opinion, then an ethics application must be submitted, although consent (written or oral), to the extent appropriate to the situation is required. For example, if the interviewees agree to be interviewed, consent may be assumed, but they should be explicitly asked if they are agreeable to be interviewed about the subject before the interview begins (see [[Article 10.2 of TCPS2|http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/chapter10-chapitre10/#toc10-intro]]). If possible and with permission of the interviewee, this question and response should be recorded.
>
>Describe a consent process that ensures that the interviewee is fully informed. This may involve written or oral consent. Where oral consent is appropriate, the researcher should make a contemporaneous journal entry of the event and circumstances or audio recording of the event.
>
>''6.6 Obtaining Consent'', p. 24:
>
>Article [[10.2 of TCPS2|http://www.pre.ethics.gc.ca/eng/policy-politique/initiatives/tcps2-eptc2/chapter10-chapitre10/#toc10-intro]] also states that 岠a variety of circumstances, signed written consent is not appropriate in qualitative research峥archers wishing to obtain consent through means other than written consent must provide an explanation to the board in section 6.6.A and describe the alternative means of obtaining and documenting consent.
*SIGGRAPH Asia review / setting up account / download materials
*setting up SVN for Overview
**ensuring Overview data still workable
*domestic interruptions journal paper
!!15 June
*website updates
*domestic interruptions journal paper
*UDLS
!!References
<<bibliography>>
!!18 June
*Domestic interruption journal paper editing
*Overview Data analysis: [[Overview User 01]]
**only 5 days of analysis w/ Overview: May 01, 02, 09, 17, 18 - substantial use only on May 01, 02,
**2,309 unique documents (out of 6,000) viewed;
**12 documents opened in the browser (one document twice)
*Overview demo to UBC chapter of VIVA VA challenge group
**~10 students in attendance
**demoed Overview w/ DTES and Vancouver City Council datasets
**not currently doing unstructured text data analytics
**forthcoming projects with the Ubyssey, Vancouver Sun
**currently using Inspire, [[CZSaw|http://czsaw.iat.sfu.ca/]], [[Jigsaw|http://www.cc.gatech.edu/gvu/ii/jigsaw/]], [[ManyEyes|http://www-958.ibm.com/software/data/cognos/manyeyes/]]
**shared download links, correspondence
!!19 June
**following up on analysis of [[Overview User 01]] log file
**meeting visiting U. Canterbury (NZ) researchers
**LUNCH (missed due to lunch w/ NZ guys)
**Tuesday Tea (cancelled?)
**[[InfoVis in the Calgary Herald: Fat Fonts|http://www.calgaryherald.com/technology/scientists+develop+font+that+helps+simplify+complex/6797475/story.html]] / http://fatfonts.org/
!!20 June
*(birthday!)
*reading for ~InfoVis:
**[[Persuasive Games: Exploitationware|http://www.gamasutra.com/view/feature/134735/persuasive_games_exploitationware.php?print=1]] (Ian Bogost, PhD in comparative literature and is director of Georgia Tech⡤uate program in digital media)
>//neral practice of extracting personal information from customers by pretending that one's product is actually one's customer. Google and Facebook's seemingly free services also could be called exploitationware of a different kind, since they use the carrot of free services (their purported product) to extract information that forms the real basis for their revenues (their real product)//
**[[GAMIFICATION IS BULLSHIT: My position statement at the Wharton Gamification Symposium|http://www.bogost.com/blog/gamification_is_bullshit.shtml]] (Ian Bogost)
>//馩cation is marketing bullshit, invented by consultants as a means to capture the wild, coveted beast that is videogames and to domesticate it for use in the grey, hopeless wasteland of big business, where bullshit already reigns anyway//
**[[The Curse of Cow Clicker: How a Cheeky Satire Became a Videogame Hit|http://www.wired.com/magazine/2011/12/ff_cowclicker/all/1]] (Wired)
>//Bogost榥rings駮ed under the auspices of his small development company, Persuasive Games䠴o simulate grinding, unsatisfying everyday experiences.//
>
>//[Zynga]᭥s were 堢ehaviorist experiments with rats�alling them סll Street hedge-fund guys of games//"
>
>//A Slow Year is a series of what Bogost calls 堰oems,�minigames in which players accomplish leisurely, pensive tasks, like slowly sipping a cooling cup of coffee or focusing on a twig as it bobs down a stream. Bogost spent three years sporadically working on the collection for the archaic Atari 2600, which he says forced him to accept constraints similar to those self-imposed by Imagist poets, like Ezra Pound, who tried to use the most precise language possible in their work. If Cow Clicker is Bogost顧nosis of what games shouldnﬠA Slow Year is his vision of what they might aspire to쯲ing the artistic frontiers of the medium to create new kinds of experiences//
>
>//鮥sses can employ new cow-clicking mechanics such as clicking a cow to distract customers from the vapid pointlessness of their products and services.꾃-TOC meeting
*Domestic interruption journal paper
*MUX meeting
*~InfoVis group meeting
*email from JS re: analysis
**missing log file from JW
!!21 June
*ML/Stats papers from ~InfoVis group
**Also: [[Alice Marwick - Status Update: Celebrity, Publicity & Self-Branding in Web 2.0|http://www.tiara.org/dissertation/index.html]]
*Overview log file analysis
*meeting w/ TM + JS via Skype: [[minutes|TM-JS-12.06.21]]
*Domestic interruption journal paper analysis
*~InfoVis group: beers at Alibi room 7:30
!!25 June
*Overview user #1 log file data analysis
!!26 June
*Overview user #1 log file data analysis
*errands in the afternoon
!!27 June
*1st Overview user wrote a [[PBS blog post about his process|http://goo.gl/hs5uW]]
*reading <<cite Kandel2012a bibliography:Bibliography>> for ~InfoVis reading group
*no IMAGER social today, no MUX
*SIGGRAPH Asia review
*Overview user #1 log file data analysis - pruning
*~InfoVis reading group
!!28 June
*Overview user #1 log file data analysis - filtering, pruning, doc reading
!!29 June
*Domestic interruption journal paper
**JM: revert to 3-group analysis, 1-group analysis not interesting enough
**results figure editing
*Overview user #1 log file data analysis - finishing up
*UDLS
!!References
<<bibliography>>
!!02 July
*(Canada day holiday)
!!03 July
*(in Leavenworth, WA)
!!04 July
*Domestic interruptions journal paper
**TRL data analysis
**figure editing
**paper revision
*MUX
**J. Cooperstock
!!05 July
*Domestic interruptions journal paper revisions
!!06 July
*Domestic interruptions journal paper revisions / writing
*SIGGRAPH Asia review
*UDLS
!!References
<<bibliography>>
!!09 July
*car service
*Education amount tax review issue
*SIGGRAPH Asia review
*Overview user #1data analysis and presentation writing
!!10 July
*Overview user #1data analysis and presentation writing
*reading <<cite Dasgupta2012 bibliography:Bibliography>> for ~InfoVis reading group
!!11 July
*Overview user #1data analysis and presentation writing
!!12 July
*(camping in Banff, AB)
!!13 July
*(camping in Banff, AB)
!!References
<<bibliography>>
!!16 July
*(camping in Banff, AB)
!!17 July
*(camping in the Okanagan)
!!18 July
*reading TM book chapter for ~InfoVis group meeting
*DRITW next steps meeting w/ MS, TM, SI
*MUX forum
**SN on peer-review system survey results
**IK on rhythm susceptibility
*~InfoVis group meeting: TM book chapter 13 on analysis
*Overview user #1data analysis and presentation writing
!!19 July
*Overview user #1data analysis and presentation writing
*analysis of extant texts: usage notes, email threads, blog post
*meeting w/ TM and JM: [[minutes|TM-JM-12.07.19]]
*request meeting w/ JS, 11am Thursday 26th
>//Given that it's been a month since we last spoke, would you like to meet next week? I can give an update regarding Overview usage analysis, while you can give us an update regarding Overview v2's development. We can then discuss next steps in terms of interviewing users: I noticed 2 more users added to the Google Doc. It might also be worth scheduling regular monthly update meetings between the 3 of us.//
**JS response:
>//Briefly, there are two more users who have a specific document set they want to apply Overview to. Both were getting a little hung up in the document import stage, so this week I put together a script that will load a directory full of ~PDFs in one shot. One is a reporter with some sensitive and docs, another seems to be a datavis professional who has some conference papers she wants to look at (which makes me sad, because document corpus research on academic papers always makes me sad; they're really unrepresentative.) There are a few additional folks playing around with the tool but I have not heard that they want to use it on a specific document set.//
>
>//We are aiming to have an early release of the web version of Overview, a minimal milestone we are calling Overview 0.1, to show at the Online News Association conference in San Francisco, Sept 20-22. I will be there and probably also our front end dev (and part time journalist) Adam Hooper. We are planning a significant push for new users at that time.//
>
>//Your point about ~DocumentCloud being a possible barrier is a good one. It will go away in time but for the moment requiring ~DocumentCloud is the easiest path to get something one this deadline, and shouldn't be a problem for our professional users since ~DocumentCloud is pretty liberal with logins for anything approximately the shape of a news organization. Actually I'm sure Ted Han would create logins for arbitrary people anyway.//
>
>//Your slide deck is awesomely detailed. This is great information, but I think you are right to ask what questions we can really answer from this. I'll have to take a more thorough look at this and see what we might discuss. Your questions about process and comprehensiveness of reading are really interesting. He at least looked at a much higher fraction of the material than I would have guessed.//
!!20 July
*(friends visiting)
!!References
<<bibliography>>
!!23 July
*Overview project
**2 new users added to [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]]
**user #1 data analysis: coding extant texts, corroborating, refining findings slide deck
**refining [[interview foci and questions|Overview Deployment Field Survey]]
**literature search: sensemaking and tagging, personal information management in visualization, incremental tagging vs. one-shot en-masse tagging, structured vs. unstructured information
**overview logging seems to be broken?
*reviewing TM's textbook draft
*reading:
**<<cite Budiu2009 bibliography:Bibliography>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
!!24 July
*resolving Overview logging issue
*writing literature review notes
*organizing cabinet
*LUNCH
*reading:
**<<cite Grimmer2011>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Marchionini2006>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
!!25 July
*testing Overview logging fix
*reading:
**<<cite Wagstaff2012>>: ML that matters - [[notes|Information Visualization Evaluation: Meta-Analysis]]
*CTOC group meeting 11am
*IMAGER pizza social
*JS Nieman Lab post: [[Who should see what when? Three principles for personalized news|http://www.niemanlab.org/2012/07/who-should-see-what-when-three-principles-for-personalized-news/]]
*Safari update, browser incompatibility w/ TiddlyWiki issue, downloaded FireFox
*reading:
**<<cite Heer2007a>>: asynchronous collaboration and visualization: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*meeting w/ ~InfoVis group 3pm: TM
*meeting w/ TM, T. M⬠A. Sarkar re: text visualization 4:30pm.
!!26 July
*meeting w/ TM + JS re: Overview dev plan, studies 11am: [[minutes|TM-JS-12.07.26]]
*CSGSA orientation meeting ICICS-204 3pm
*planning [[EPSE 595]] [[presentation for MUX forum|http://prezi.com/ewtzqi2qsjbk/hci-and-qualitative-research-methods-or-why-mux-students-should-take-epse-595/]]
**course outline (vs. SOCI 503)
**reading list: [[Bibliography-EPSE595]]
**assignments / workbooks: personal epistemology / mind maps, observation, interview, found data, data analysis
**in-class activities: discussing Pascoe, interviewing, material data analysis, data analysis, computer-assisted data analysis tools
**methodologies: [[Grounded Theory]], [[Ethnography]], [[PhotoVoice|http://prezi.com/_m_lndsuctib/photovoice/]], [[Narrative Analysis|http://prezi.com/i9c0uupsydvi/narrative-analysis/]], [[Action Research]]. [[Phenomenolgy|http://prezi.com/gudzqww9jcv8/phenomenology/]]
*considering next ~InfoVis discussion slot (Aug 15):
**paper or tagging/flagging/PIM discussion
**RPE pre-paper talk?
!!27 July
*(in Ottawa)
!!References
<<bibliography>>
!!13 August
*(7am flight to Vancouver, in in the afternoon)
*catching up on email (100 unread), LUNCH FB, twitter
*CSGSA orientation details
*considering next ~InfoVis discussion slot (Aug 29, switched w/ Stephen):
**paper or tagging/flagging/PIM discussion
**RPE pre-paper talk?
*interesting links:
**via OS: [[Interesting take on gamification and augmented reality; a bit of a speculative warning|http://vimeo.com/46304267]]
**via SI: [[NYTimes visualization on Olympic 100m sprinters|http://www.nytimes.com/interactive/2012/08/05/sports/olympics/the-100-meter-dash-one-race-every-medalist-ever.html]]
**via K. Murhpy: [[The Ph.D. Grind: Candid Discussions About Ph.D. Life|http://pgbovine.net/PhD-memoir.htm]]
*reading:
**<<cite Wood2011 bibliography:Bibliography>> for ~InfoVis: [[notes|Information Visualization: Design Studies]]
***Many bills: a visual bill explorer: [[http://manybills.researchlabs.ibm.com/]]
***mySociety: [[http://theyworkforyou.com/]]
***[[http://onomap.org/]]
!!14 August
*Journal writing: on creativity
*planning [[EPSE 595]] [[presentation for MUX forum|http://prezi.com/ewtzqi2qsjbk/hci-and-qualitative-research-methods-or-why-mux-students-should-take-epse-595/]]
**course outline (vs. SOCI 503)
**reading list: [[Bibliography-EPSE595]]
**assignments / workbooks: personal epistemology / mind maps, observation, interview, found data, data analysis
**in-class activities: discussing Pascoe, interviewing, material data analysis, data analysis, computer-assisted data analysis tools
**methodologies: [[Grounded Theory]], [[Ethnography]], [[PhotoVoice|http://prezi.com/_m_lndsuctib/photovoice/]], [[Narrative Analysis|http://prezi.com/i9c0uupsydvi/narrative-analysis/]], [[Action Research]]. [[Phenomenolgy|http://prezi.com/gudzqww9jcv8/phenomenology/]]
*LUNCH 12pm x836
*[[notes|Information Visualization: Design Studies]] for <<cite Wood2011 bibliography:Bibliography>> for ~InfoVis
*pre-paper talk brainstorming.
*reading:
**<<cite Ziemkiewicz2012>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
!!15 August
*writing on exploratory curricula in education
*reading:
**<<cite Pinaud2012>>: [[notes|Information Visualization: Design Studies]]
**<<cite Kairam2012>>: [[notes|Information Visualization: Techniques]]
*MUX meeting 2pm x836: AF ~MSc presentation
*meeting w/ ~InfoVis group 3pm: JD: Discussing <<cite Wood2011>>
!!16 August July
*writing
*commenting on [[Twheel|http://twheel.com/]]'s blog post about radial layouts
*meeting w/ TM + JM: [[minutes|TM-JM-12.08.16]]
*Research pitch to Tableau (end of August)
**research interests, background, project ideas
>''abstract'': Mixed-method evaluation of visualization tools and techniques
>''current interests'': Exploratory data analysis, serendipitous discovery; High-dimensional data, DR data, unstructured text data
>''emerging methodologies'': Crowdsourcing social data analysis, graphical inference
>''ongoing interest'': Graphical Perception, Visual Attention, ~Task-Switching and Interruptions
>''background'': HCI M.Sc specialization supervised by J. ~McGrenere, Cognitive Science undergraduate degree, 16 month professional internship as UX designer at EMC
>''courses'': in quant. and qual. research methods, visualization, HCI, visual display design and graphical perception, scene perception, data mining
*reference management: added CPSC 533C and PSYC 579 references to [[References: Read]]
!!17 August July
*writing
*Research pitch to Tableau (end of August) - draft:
>My research is about furthering methodologies for evaluating visualization tools and techniques. This involves executing, comparing, and writing about quantitative and qualitative methods used for determining whether a visualization tool or technique is usable, useful, and appropriate for its deployment context. Another aspect of this work is determining at which points these methods are appropriate and informative during the various phases of tool or technique development. Finally, this work will allow me to explore the potential of emerging evaluation methods, such as those involving crowdsourcing or Wickham et al's graphical inference technique.
>
>Of particular interest to me is how we can design evaluation methodologies that assess how visualization tools support exploratory data analysis and serendipitous discovery in large datasets. My most recent project is an example of this: a mixed-method evaluation of Overview, a tool for visualizing and exploring large text document sets, built by our collaborators at the Associated Press. My ongoing objective will be to compare the efficacy of evaluation methods used in the Overview project with those used in future projects.
>
>I came to be interested in this area or research during my undergraduate studies in cognitive science, where I was fascinated by graphical perception, working memory, and attention, along their implications for visual display design. This interest persisted throughout my Master's work in HCI, where I focused on aspects of task-switching and interruptions relevant to interface design. It was during this time that I began to realise the limitations of quantitative methodologies for answering many research questions, and I acknowledged a need for the triangulation of research methods, both quantitative and qualitative, those deployed in controlled settings as well as those deployed "in the wild".
>
>I think that it's an exciting time to be carrying out this work in the visualization domain. As demand for new visualization techniques and tools continues to grow, visualization practitioners and researchers will need to adopt evaluation methodologies that provide the best answers to questions about visualization usability and utility.
*reference management
*RPE writing
*reading:
**<<cite Heer2010>>: [[notes|Information Visualization Evaluation: Meta-Analysis]] - crowdsourcing graphical perception
!!References
<<bibliography>>
!!20 August
*writing
*orientation week planning: hike and bowling
*writing RPE paper draft
*reading:
**[[Using Word Clouds for Topic Modeling Results|https://dhs.stanford.edu/algorithmic-literacy/using-word-clouds-for-topic-modeling-results/]] (Stanford Digital Humanities' E. Meeks, via N. de Freitas)
>//ᣴually shows a lack of understanding of the primary principal of data visualization to condemn a form, and though it makes for better copy to declare word clouds to be �l�y that ﳴ nothing�d be shown as a network, it塬ly the application and not the method that is being criticized. Word clouds, for instance, are really good when paired with topics generated from topic models槴;
>//ﲠand rotation are arbitrary in Wordle and typically in word clouds, but they need not be. Color could be used to indicate part-of-speech, and rotation could be used to indicate named entity type. Or, they could be used to reinforce incidence of the wordꪎYT's Jacob Harris on [[Word Clouds|http://www.niemanlab.org/2011/10/word-clouds-considered-harmful/]] via Niemen Lab
>//୵llets of the Internet寴;Every time I see a word cloud presented as insight, I die a little inside"//
>
>ᵯt;Visualization is reporting, with many of the same elements that would make a traditional story effective"//
**<<cite Kosara2010 bibliography:Bibliography>>: [[notes|Information Visualization Evaluation: Meta-Analysis]] on conducting ~InfoVis evaluation studies with [[MTurk|http://mturk.com/]]
***links: http://www.thesheepmarket.com/
***[[Turkit|http://groups.csail.mit.edu/uid/turkit/]]
***[[CrowdFlower|http://crowdflower.com/]]
***[[J. Ross, A. Zaldivar, L. Irani, and B. Tomlinson. Who are the turkers? worker demographics in amazon mechanical turk.|http://www.ics.uci.edu/ﳳ/pubs/SocialCode- 2009-01.pdf]]
*term scheduling
*email re: older participants (C. Neustaedter)
*[[BELIV '12 Workshop|http://www.beliv.org/wiki/BELIV2012]]
!!21 August
*RPE writing
*LUNCH meeting
!!22 August
*writing
*RPE writing
*reading:
**<<cite Beaudouin-Lafon2004>>: [[notes|Evaluation in HCI: Meta-Analysis]] - AVI paper on designing interaction, not interfaces
!!23 August
*Request from MH: Killam mentoring award for JM (letter by Aug 31)
*writing
*writing RPE paper
!!24 August
*writing
*revising and completing [[RPE paper draft|http://goo.gl/xH6Vu]]: Evaluation and Visualization for Exploratory Data Analysis: A Case Study in Journalism
*reading:
**<<cite Wickham2010b>>: [[notes|Information Visualization: Techniques]] - layered grammar of graphics
!!References
<<bibliography>>
!!27 August
*writing
*[[notes|Information Visualization: Techniques]] for <<cite Wickham2010b bibliography:Bibliography>> (layered grammar) and [[notes|Evaluation in HCI: Meta-Analysis]] for <<cite Beaudouin-Lafon2004>>
*reading:
**<<cite Wickham2010>> (re-read): [[notes|Graphical Inference Evaluation]]
**<<cite Buja2009>> - longer journal version of graphical inference for EDA / MD: [[notes|Graphical Inference Evaluation]]
*CSGSA orientation week event planning
*term scheduling
*email from Peter S. Jenkins, Assoc. Prof @ York U, re: ~WikiLeaks cables
*~InfoVis group meeting paper discussion this week: <<cite Willett2012>> (crowdsourcing)
!!28 August
*writing
*Orientation week event planning
*notes for <<cite Buja2009>> and <<cite Wickham2010>>
*reading:
**<<cite Pohl2010>>: [[notes|Information Visualization Evaluation: Qualitative Methods]] - log file analysis of EDA
**<<cite Willett2012>> (crowdsourcing, re-read for ~InfoVis reading group)
*LUNCH
*study piloting w/ JD
*fiddling w/ R.
!!29 August
*Orientation week event planning - UDLS / bowling
*reading:
**<<cite Willett2012>> (crowdsourcing, re-read for ~InfoVis reading group) + [[notes|Information Visualization Evaluation: Qualitative Methods]]
*skimming:
*<<cite Zhao2012>>, <<cite Majumder2010>> - Iowa State Tech Reports on graphical inference
*Overview: email from academic user using Overview to mine publication abstracts
*LUNCH/SPIN lab space meeting 10am x508
*MUX 2pm - OS ~MSc presentation
*~InfoVis meeting 3pm (discussing <<cite Willett2012>>)
*fiddling w/ R.
*IMAGER beach BBQ 5pm - Locarno beach
!!30 August
*renewing ACM student membership; registering for ~VisWeek 2012
*writing on user-centred design
*meeting w/ TM + JM: [[minutes|TM-JM-12.08.30]]
*reference mgmt.
*piloting experiment w/ SH
*letter fro Killam mentoring prize
!!31 August
*reference mgmt (cont.)
*letter fro Killam mentoring prize (cont.)
*MUX space work party
*RPE revisions
!References
<<bibliography>>
!!03 September
*(labour day)
!!04 September
*paying registration fees
*writing
*RPE revisions
*reviewing SI's SIGRAD paper draft for ~InfoVis
*reviewing SN's CHI paper drafts for MUX
*LUNCH
*car insurance renewal
!!05 September
*~InfoVis group meeting x530 10am - SI SIGRAD paper draft
*meeting w/ TM 11am: [[minutes|TM-12.09.05]]
*MUX - 2pm SN CHI paper draft
*RPE revisions
*orientation course pitch 6pm
*pizza / board games
!!06 September
*RPE revisions
*contacting JM @ Tableau re: internship chat
*CPSC 508 (DMP 101, 11-12:30)
*bowling night 8pm
!!07 September
*contacting RR re: RPE scheduling
*reading CPSC 508 papers
*orientation receipts
*UDLS
*beer call
!!10 September
*[[CPSC 508]] paper reviews
*writing
*~InfoVis meeting 2 - 3:30pm in ICICS/CS 204: website design discussion
*RPE revisions
!!11 September
*RPE revisions
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH meeting 12:30 in x860
*reading:
**<<cite Meyer2012 bibliography:Bibliography>> BELIV submission: [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Heer2012a>>: [[notes|Task Characterization: Meta Analysis]]
*Tuesday Tea
*[[CPSC 508]] readings
!!12 September
*[[CPSC 508]] paper reviews
*responding to P. Jenkins (Overview user): troubleshooting
*reading MH CHI paper draft for MUX
*meeting w/ TM: [[minutes|TM-12.09.12]] 3-4pm
*RPE session chair recruiting
!!13 September
*RPE session chair recruiting
*responding to P. Jenkins (Overview user): troubleshooting - Overview not installed?
*reading:
**<<cite Yi2007>> - Interaction in ~infoVis taxonomy: [[notes|Task Characterization: Meta Analysis]]
**<<cite Zhou1998>> - visual task taxonomy in visual discourse applications: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] lecture DMP 101 11am
!!14 September
*[[CPSC 508]] readings
*UDLS
*reading:
**
!!References
<<bibliography>>
!!17 September
*[[CPSC 508]] paper reviews
*reading:
**<<cite Kandel2012 bibliography:Bibliography>>: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*~InfoVis meeting 2 - 3:30pm in ICICS/CS 104: discussing <<cite Kandel2012a>>
*meeting w/ MS: [[DRITW|HDD-DR Ethnographic Project]] next steps: week following ~VisWeek
*RPE chair found (R. Garcia), Fri Sep 28 4-5pm, room TBD
!!18 September
*reading:
*sent out RPE presentation invitation
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH meeting 12:30 in x860
*reading:
**<<cite Amar2005>>: [[notes|Task Characterization: Meta Analysis]] (re-read from CPSC 533C F09)
**<<cite Chi1998>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Wehrend1990>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Roth1990>>: [[notes|Task Characterization: Meta Analysis]]
*Tuesday Tea
*[[CPSC 508]] readings
!!18 September
*no MUX meeting (CHI deadline)
*meeting w/ JM: (postponed due to CHI)
*meeting w/ AT&T's C. Sheidegger
*reading:
**<<cite Morse2000>>: [[notes|Task Characterization: Meta Analysis]]
*seeking ~VisWeek West Coast party venue
!!20 September
*seeking ~VisWeek West Coast party venue
*writing
*re-reading:
**<<cite Shneiderman1996>>: [[notes|Task Characterization: Meta Analysis]] (re-read)
*[[CPSC 508]] lecture DMP 101 11am
*meeting w/ JM 2pm: [[minutes|JM-12.09.20]]
!!21 September
*seeking ~VisWeek West Coast party venue
*reading:
**<<cite Lee2006>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Lam2008a>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Card1999>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] readings
*UDLS
*MUX party 7pm @ TM's house
!!References
<<bibliography>>
!!24 September
*[[CPSC 508]] paper reviews
*RPE presentation prep
*~InfoVis meeting 2 - 3:30pm in ICICS/CS 104: MS ~VisWeek practice talk
!!25 September
*RPE presentation prep
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH meeting 12:30 in x860
*Tuesday Tea
*[[CPSC 508]] readings
*reading:
**<<cite Jankun-Kelly2007 bibliography:Bibliography>>: [[notes|Task Characterization: Meta Analysis]]
!!26 September
*[[CPSC 508]] paper reviews
*no meeting w/ TM - task taxonomy update instead:
>I don't think it's urgent to meet tomorrow, so instead I thought I'd send an update on the task taxonomy literature review and the [[annotated bibliography|Task Characterization: Meta Analysis]] I'm maintaining. I've expanded my scope to task characterizations in the domains of information retrieval and human-information-interaction; JD helped point me out some good places to start. I still have lots to read, and at some point soon I'll commence sorting and categorizing task taxonomies (or subsets of task taxonomies).
*TM response:
>//Looks very good, we don't need to chat yet. My list of things that I should really read is: [Klahr1999] - Meta-Review of Scientific Discovery, [Marchionini2006] - Searching vs. Browsing. Also in a perfect world before Oct 13 I would read all the BELIV10 papers and all the BELIV12 papers so that would include the Mayr paper in that sweep. I'll see if I can scrounge up the time for that or not...//
*no MUX meeting
*IMAGER social
*reading:
**<<cite Casner1991>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Gotz2008>>: [[notes|Task Characterization: Meta Analysis]]
!!27 September
*reading:
**<<cite Ware2004>>: (re-read on tasks) [[notes|Task Characterization: Meta Analysis]]
**<<cite Mullins1993>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] lecture DMP 101 11am
*RPE prep
!!28 September
*reading:
**<<cite Tory2004>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] readings
*RPE presentation in ICCS 304 - 4pm
*UDLS
!!References
<<bibliography>>
!!01 October
*[[CPSC 508]] paper reviews
*response to GC re: [[RPE paper|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]]
*addressing Ron's [[RPE comments|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]]
**booked meeting Wed 10am in iccs-204
*Overview user: M. Conroy on tweets:
>//"I'd be happy to help. Overview is a great project and I'm keen to see it grow. What's next? :)//"
*considered attending VIVA open house 12-1 Irving K. Barber learning centre, Dodson Room (did not attend)
*~InfoVis meeting 2 - 3:30pm in ICICS/CS 104: MS ~VisWeek practice talk
*(car issues)
*reading:
**<<cite Chuah1996 bibliography:Bibliography>>: [[notes|Task Characterization: Meta Analysis]]
!!02 October
*addressing Ron's [[RPE comments|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]]
*[[Overview users Google group|https://groups.google.com/forum/#!forum/overview-users]] launched
*reading:
**<<cite Plaisant1995>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH meeting 12:30 in x860
*Tuesday Tea
*[[CPSC 508]] readings
!!03 October
*[[CPSC 508]] paper reviews
*meeting w/ TM, RR: re: [[RPE comments, suggestions, discussion|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]] - 10am in iccs-204 (JM absent)
*literature search
*MUX meeting: DT thesis talk
*reading:
**<<cite Roth2012>>: [[notes|Task Characterization: Meta Analysis]]
!!04 October
*lecture on future of academic publishing @ IKB 302
**[[Planned Obsolescence: Publishing, Technology, and the Future of the Academy|http://t.co/3e2Z0sH0]] - [[Kathleen Fitzpatrick|http://www.plannedobsolescence.net/]] ([[@kfitz|https://twitter.com/kfitz]]) - U. Cal. Claremont
***author of //Anxiety of Obsolescence: The American Novel in the Age of Television//
***the //cultural wildlife preserve// of the academic book
***post-dot-com crash and academic publishing, unsustainable economic model - yet the book/monograph required for tenure/promotion (esp. in humanities) - academic books are undead - avg. monograph sells 400 copies in its lifetime
***content nor form is the issue (technological change is inevitable), it's the distribution system (requires institutional, intellectual, social change): digital conversion will not solve the problem. "But we've never done it that way before"
***[[MediaCommons|http://mediacommons.futureofthebook.org/]]
***addressing peer review models (and the psychology of peer review): scarcity of time/attention/effort, a need for regulation, communication, visibility (open/closed, anonymous/identifiable), metrics of peer review (hits, downloads, inbound links, comments)
***implications for citation management, co-authorship
***works/authors cited: Roland Barthes, //Infotopia//, //Enemies of Promise// (Waters), //Access Principle//, //Googlization of Everything//, //Scholarship and the Digital Age//
*reading:
**<<cite Tukey1980>>: [[notes|Task Characterization: Meta Analysis]]
**<<cite Shrinivasan2008>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] lecture DMP 101 11am
!!05 October
*(at home, leaving for Portland midday)
*reading:
**<<cite Dou2009>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] readings
*UDLS
!!References
<<bibliography>>
!!08 October
*(Thanksgiving holiday)
*[[CPSC 508]] paper reviews
!!09 October
*CPSC discussion prep
*reading:
**<<cite Rensink2012 bibliography:Bibliography>>: [[notes|Task Characterization: Meta Analysis]]
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH meeting 12:30 in x860
*Tuesday Tea
*[[CPSC 508]] reading / paper review
!!10 October
*[[CPSC 508]] paper review
*floor warden meeting 10am iccs 238
*meeting w/ TM on [[Task Characterization: Meta Analysis]]: [[minutes|TM-12.10.10]]
*reading:
*Skype chat w/ Rob re: visualizing ~SNPs
!!11 October
*reading:
*~InfoVis group meeting: TM ~VisWeek panel discussion practice talk
*[[CPSC 508]] lecture DMP 101 11am
*MAGIC open house 4:30pm
!!12 October
*Overview user story: [[Ryan asked for federal help as he championed cuts|http://www.cbsnews.com/8301-505245_162-57531247/ryan-asked-for-federal-help-as-he-championed-cuts/]], supplemental [[annotated documents|https://www.documentcloud.org/documents/459102-paul-ryans-letters-to-executive-agenices.html]] in ~DocumentCloud
*[[CPSC 508]] paper proposal, paper reading
*UDLS
!!References
<<bibliography>>
!!14-15 October
*[[BELIV 2012 Workshop|BELIV-12: Notes]] - Seattle WA
**[[References|References: BELIV]] [[BibTeX|BELIV-12]]
!!16-19 October
*[[VisWeek|VisWeek12 Notes]] - Seattle WA
!!22 October
*[[CPSC 508]] paper reviews
*[[DRITW|HDD-DR Ethnographic Project]] re-writing
*~VisWeek expense reporting
*Meeting w/ MS 2pm re: [[DRITW|HDD-DR Ethnographic Project]]
*~InfoVis group meeting cancelled
!!23 October
*[[CPSC 508]] lecture DMP 101 11am
*Meeting w/ MS 4pm re: [[DRITW|HDD-DR Ethnographic Project]]
*[[CPSC 508]] reading / paper review
!!24 October
*[[CPSC 508]] paper review
*Imager social:
**LO on interacting with radiology images
**MB on 3D surface rendering from 3D line sketches
*MUX:
**VR ~MSc presentation on pointing, multiple object tracking for large-screen displays
*[[DRITW|HDD-DR Ethnographic Project]] re-writing
!!25 October
*[[DRITW|HDD-DR Ethnographic Project]] re-writing
*[[CPSC 508]] lecture DMP 101 11am
!!26 October
*[[DRITW|HDD-DR Ethnographic Project]] re-writing
*meeting w/ TM, MS re: [[DRITW|HDD-DR Ethnographic Project]] 1:30pm
*CHI review
*~VisWeek references
*[[CPSC 508]] paper reading
*UDLS
!!29 October
*[[CPSC 508]] paper reading / reviews
*GRAND reporting
*~InfoVis group meeting: ~VisWeek recap, website, term scheduling
*[[Task Characterization: Meta Analysis]]: adding ~VisWeek papers
!!30 October
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH 12:30 - (GRAND reporting?)
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
*Tuesday tea
!!31 October
*[[CPSC 508]] paper review
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
!!01 November
*CPSC 344 guest lecture (CHI talk)
*[[CPSC 508]] lecture DMP 101 11am
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
!!02 November
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
*MAGIC talk:
**Liane Gabora on "Computational Modeling and Interactive Visualization of Creativity and Cultural Evolution."
***psychology of creativity, Darwinian creativity, honing theory,
***evolutionary creativity: auto-poetic, self mending, self-organizing, Barron (1963) on 2 sides of creativity, Newell & Simon (1962) on modelling creativity
***insight = self-organized criticality
***balance between numbers and performance of creators and imitators predicts overall fitness of the system
*[[CPSC 508]] paper reading
*UDLS
!!05 November
*[[CPSC 508]] paper reading / reviews
*~InfoVis group meeting: dicussing <<cite Anand2012 bibliography:Bibliography>>
*[[Task Characterization: Meta Analysis]]: adding ~VisWeek papers
*(at a concert)
*CHI review follow-up
!!06 November
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH 12:30 (optional, JM away)
*DRITW revisions
!!07 November
*meeting w/ TM + JM: [[minutes|TM-JM-12.11.07]]
*[[CPSC 508]] paper review
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
*MUX: SK on tablet ~PCs in education, (CIRS decision theatre)
*meeting w/ MS re: DRITW editing
!!08 November
*[[CPSC 508]] lecture DMP 101 11am
*[[Task Characterization: Meta Analysis]]: preparing pre-pre-paper slide deck
*[[CPSC 508]] paper reading / reviews
*DLS: Judea Pearl (UCLA, Turing award winner) ESB room 1013 - on the mathematics of cause and effect
!!09 November
*met with MF re: CPSC 508
*[[Task Characterization: Meta Analysis]]: finding forward references
*[[CPSC 508]] project
*UDLS
!!References
<<bibliography>>
!!13 November
*(Remembrance day holiday)
*[[CPSC 508]] paper review + discussion prep
!!13 November
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH 12:30 - 1:30
*emailing MF re: CPSC 508
*MUX talk on [[EPSE 595]]
!!14 November
*meeting w/ TM + MS re: DRITW
*meeting w/ TM: [[minutes|TM-12.11.14]]
*email intro from RR: re Seattle Times' [[Eric Ulken|http://ulken.com/]]
*emailing JS re: Overview + AP / Paul Ryan correspondence story
*[[CPSC 508]] paper review + discussion prep
!!15 November
*[[CPSC 508]] discussion prep, lecture DMP 101 11am
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*~InfoVis group party @ TM 6-9pm
!!16 November
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*[[CPSC 508: Project]] reading/writing
*UDLS
!!References
<<bibliography>>
!!19 November
*[[CPSC 508]] paper review, reading
*~InfoVis group meeting 2pm: SI's Glint SIGRAD practice talk
*emailing AP's JG on setting up an Overview interview
!!20 November
*[[CPSC 508]] lecture DMP 101 11am
*LUNCH 12:30 - 1:30
*Overview: jg_ryan_foias log file analysis
!!21 November
*Overview: interview booked with JG Dec 5
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*[[CPSC 508]] paper review
*meeting w/ TM: [[minutes|TM-12.11.14]]
*MUX talk on [[EPSE 595]]: [[slides|http://prezi.com/ewtzqi2qsjbk/hci-and-qualitative-research-methods-or-why-mux-students-should-take-epse-595/?auth_key=747a37685f3c46674cebb40b96668a58bb4a77b8]]
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
!!22 November
*[[CPSC 508]] lecture DMP 101 11am
*[[CPSC 508: Project]] reading/writing
!!23 November
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*UDLS
!!References
<<bibliography>>
!!26 November
*[[Task Characterization: Meta Analysis]]: consolidating and mapping (in x5 hallway)
*[[CPSC 508]] paper review, reading
*~InfoVis group meeting 2pm: reading <<cite Ma2012 bibliography:Bibliography>>
!!27 November
*Overview jg_foias_log file analysis
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*[[CPSC 508]] lecture DMP 101 11am
*read <<cite Pohl2012a>> on high-level theories: [[notes|Task Characterization: Meta Analysis]]
!!28 November
*MUX whiteboard specs
*[[Task Characterization: Meta Analysis]]: consolidating and mapping
*IMAGER social
*[[CPSC 508]] paper review
*meeting w/ TM: [[minutes|TM-12.11.28]]
!!29 November
*[[CPSC 508]] lecture DMP 101 11am
*[[CPSC 508: Project]] reading/writing
!!30 November
*meeting w/ TM: [[Task Characterization: Meta Analysis]]: consolidating and mapping
*[[CPSC 508: Project]] reading/writing
*UDLS
!!References
<<bibliography>>
!!03 December
*CHI SV position awarded
*~InfoVis group meeting 10am: TM [[FODAVA|http://fodava.gatech.edu/]] talk
*reading: [[Multitasking makes you dumb|http://sirupsen.com/multitasking-makes-you-dumb/]], [[Media multitaskers pay mental price, Stanford study shows|http://news.stanford.edu/news/2009/august24/multitask-research-study-082409.html]] (Stanford study)
*(picking up dad from airport, attending visitation)
*prep for Overview interview
*email
*CPSC 508 course eval
!!04 December
*[[CPSC 508: Project]] reading/writing
*LUNCH (absent, attending funeral service)
*prep for Overview interview
!!05 December
*Overview interview w/ JS, JG (ryan_foias)
!!06 December
*[[CPSC 508: Project]] reading/writing
!!07 December
*DRITW editing
*UDLS
*reading:
**<<cite Liu2008 bibliography:Bibliography>> on distributed cognition
!!References/Reading:
*[[How to Live Without Irony|http://opinionator.blogs.nytimes.com/2012/11/17/how-to-live-without-irony/]] by Christy Wampole, NYT
<<bibliography>>
!!10 December
*[[CPSC 508: Project]] writing
!!11 December
*[[notes|Task Characterization: Meta Analysis]] for <<cite Liu2008 bibliography:Bibliography>> on distributed cognition
*[[CPSC 508: Project]] writing
*DRITW feedback from MS
*Overview notes from JG (ryan_foias)
*LUNCH 12:30
*Tuesday Tea
!!12 December
*[[CPSC 508: Project]] writing
*MUX lunch
*meeting w/ JM 2pm - [[minutes|JM-12.12.12]]
*M.Sc presentation - D. Toker on individual differences in ~InfoVis
!!13 December
*[[CPSC 508: Project]] writing
!!14 December
*(flying to Ottawa)
*[[CPSC 508: Project]] writing
!!References/Reading:
*[[Programming is a Pop Culture|http://raganwald.posterous.com/programming-is-a-pop-culture]] by Reginald Braithwaite [[@raganwald|twitter.com/raganwald]]
*[[Let's All Shed Tears For The Crappy Startups That Can't Raise Any More Money|http://readwrite.com/2012/12/03/lets-all-shed-tears-for-the-crappy-startups-that-cant-raise-any-more-money]] by Dan Lyons, [[ReadWrite.com|http://readwrite.com]]
*[[Saying No to College|http://www.nytimes.com/2012/12/02/fashion/saying-no-to-college.html?pagewanted=all&_r=0]] by Alex Williams, NYT
*[[How Google峩gners Are Quietly Overhauling Search|http://www.fastcodesign.com/1671425/how-googles-designers-are-quietly-overhauling-search#5]] by Sarah Kessler, [[FastCompany|http://www.fastcodesign.com/]]
*[[Top 10 Trends for Business Intelligence: Our Outlook on 2013|http://www.tableausoftware.com/about/blog/2012/12/top-10-trends-business-intelligence-our-outlook-2013-20186?elq=f7f499e6860f45a28ddf1413dbab1098]] by Elissa Fink, [[Tableau|http://www.tableausoftware.com/]]
<<bibliography>>
(in Ottawa 12.15 - 12.31)
*reading:
**<<cite Kuhn1962 bibliography:Bibliography>> on scientific revolutions
!!15-17 December
*[[CPSC 508: Project]] writing/editing
*[[Task Characterization: Meta Analysis]]: pre-paper talk edits
*reading:
**<<cite Hollan2000>> on distributed cognition: [[notes|Task Characterization: Meta Analysis]]
!!18 December
*reading:
**[[IBM reveals five innovations that will change our lives within five years|http://www.kurzweilai.net/ibm-reveals-five-innovations-that-will-change-our-lives-within-five-years]] - [[kurzweilai.net|http://www.kurzweilai.net/]]
**<<cite Kirsh2006>> on distributed cognition: [[notes|Task Characterization: Meta Analysis]]
**<<cite Crouser2012>> on affordances in VA: [[notes|Task Characterization: Meta Analysis]]
*DRITW draft reading from MS
*[[Task Characterization: Meta Analysis]]: pre-paper talk edits
!!19 December
*DRITW meeting 10am PT / 1pm ET / 7pm ECT
*meeting w/ TM 2pm PT / 5pm ET
!!20 December
*DRITW edits / token to MS
*[[CPSC 508: Project]] editing/submitting
*reading:
**<<cite Kirsh1994>> on epistemic/pragmatic actions: [[notes|Task Characterization: Meta Analysis]]
!!21 December
*DRITW update from MS, token to TM, SI
*Reading:
**[[Snow Fall: The Avalanche at Tunnel Creek|http://www.nytimes.com/projects/2012/snow-fall/#/?part=tunnel-creek]] by John Branch, [[NYT|http://www.nytimes.com/]]
*Overview web version log file review
!!References/Reading:
<<bibliography>>
(in Ottawa 12.15 - 12.31)
!!31 December
*(New Year's Eve - flying back to Vancouver)
*[[CPSC 508: Project]] migrated from ~TeX to wiki
*reading:
**<<cite Kuhn1962 bibliography:Bibliography>> on scientific revolutions
**[[Getting Started with Data Science|http://www.hilarymason.com/blog/getting-started-with-data-science/]] by Hilary Mason via @nimalan
**[[The Most Futuristic Predictions That Came True in 2012|http://io9.com/5971328/the-most-futuristic-predictions-that-came-true-in-2012]] by George Dvorsky, io9, via OS
!!01 January
*(New Year's Day)
!!02 January
*returning to UBC, regrouping
*meeting w/ TM postponed to Jan 9
*checking out the [[Hive|http://www.hivevancouver.com/page/get-workspace]]
*writing
*CV, website edits
*internship/collab ideas:
**Adobe Research: [[Mira Dontcheva|http://www.adobe.com/technology/people/san-francisco/mira-dontcheva.html]] (HCI, search), [[Wilmot Li|http://www.adobe.com/technology/people/san-francisco/wilmot-li.html]] (graphics, visualization), [[Walter Chang|http://www.adobe.com/technology/people/san-jose/walter-chang.html]] (text analytics), [[Bongwon Suh|http://www.adobe.com/technology/people/san-jose/bongwon-suh.html]] (social network analysis, HCI)
**emailed RL at SAP re: business intelligence tools
*emailing CT re: Interruptions TOCHI paper revisions + venue change
*Overview:
**web version music reviews dataset
**Follow-up w/ Seattle Times folks re: Overview?
*reading:
**[[Meet Microsoft, the world's best kept R&D secret|http://www.pcworld.com/article/2020268/meet-microsoft-the-worlds-best-kept-randd-secret.html]] by Matt Smith (~PCWorld) via @karpathy
**<<cite Kuhn1962>> on scientific revolutions
*checking out Coursera:
**[[Stanford ML|https://www.coursera.org/course/ml]]
**[[UW Data Science|https://www.coursera.org/course/datasci]]
*Ordered Hutchins' //Cognition in the Wild//
!!03 January
*writing
**[[notes|Task Characterization: Meta Analysis]] on <<cite Kuhn1962>>
*browsing Cognitive Science society journal
*began Python tutorial
*reading:
**[[10 things every journalist should know in 2013|http://www.journalism.co.uk/news/10-things-every-journalist-should-know-in-2013/s2/a551648/]] - Sarah Marshall, journalism.co.uk
**<<cite Case2008>> on [[models of information seeking|Task Characterization: Meta Analysis]] - ch. 6
**[[2012: The Year in Graphics|http://www.nytimes.com/interactive/2012/12/30/multimedia/2012-the-year-in-graphics.html]] - NYT
**[[Census Dotmap|http://bmander.com/dotmap/index.html]] by Brandon ~Martin-Anderson
*~UPass
*PDLS - A Multimodal Approach Towards Robust ~Human-Machine Interaction
**amphibious robots, human-machine collaboration / cooperation, robochat graphical instruction language
**CARIS lab collaboration, risk assessment
!!04 January
*writing
*SI, MS feedback on DRITW
*[[CHI '13 workshops|http://chi2013.acm.org/authors/call-for-participation/workshop-participants/]]
**[[evaluation methods for creativity support environments|http://ecologylab.net/workshops/creativity/]]
**[[Many People, Many Eyes: Aggregating Influences of Visual Perception on User Interface Design|http://people.seas.harvard.edu/~reinecke/manyeyes/]]
*follow-up w/ M. Conroy, Overview user
*reading:
**<<cite Case2008>> on [[theories, perspectives, paradigms|Task Characterization: Meta Analysis]] - ch. 7
**<<cite Micallef2012>> - Proc. ~InfoVis '12 on Bayesian reasoning for ~InfoVis group discussion: [[notes|Information Visualization Evaluation: Quantitative Methods]]
*UDLS
!!References/Reading:
<<bibliography>>
!!07 January
*notes for <<cite Micallef2012 bibliography:Bibliography>> - Proc. ~InfoVis '12 on Bayesian reasoning for ~InfoVis group discussion: [[notes|Information Visualization Evaluation: Quantitative Methods]]
*~InfoVis group meeting 10:30 x530
*DRITW draft sent by MS
*Overview
**JS on logging (e.g. [[my vancouver news dataset's log|https://www.overviewproject.org/documentsets/305/log-entries]]), new user story w/ Overview: [[The Best of Our Gun Debate: Readers Weigh In on Owning Firearms|http://www.thedailybeast.com/articles/2012/12/22/the-best-of-our-gun-debate-readers-weigh-in-on-owning-firearms.html]] and [[The State-by-State Breakdown|http://www.thedailybeast.com/articles/2012/12/21/the-best-of-our-gun-debate-the-state-by-state-breakdown.html]] (12.21.12)
**response from User MC: has been using Overview desktop, willing to try web version
*Joined [[HxD Vancouver|http://www.meetup.com/HXD-Vancouver/t/wm1?rv=wm1&ec=wm1]] meetup group, [[Vancouver Data Visualization|http://www.meetup.com/Vancouver-Data-Visualization/t/wm1?rv=wm1&ec=wm1]] group
*reading:
**<<cite Toms1999>> on serendipitous information retrieval
**[[Understanding bias in computational news media|http://www.niemanlab.org/2012/12/nick-diakopoulos-understanding-bias-in-computational-news-media/]] by Nick Diakopoulos on [[niemanlab|http://niemanlab.org]]
*Hutchins' //Cognition in the Wild// arrived
!!08 January
*reading:
**[[Without Human Insight, Big Data Is Just A Bunch Of Numbers|http://www.fastcompany.com/3004000/without-human-insight-big-data-just-bunch-numbers]] by Sam Ford, [[FastCompany|http://www.fastcompany.com]]
***popular science book [[The Human Face of Big Data|http://humanfaceofbigdata.com/]] by Rick Smolan, Jennifer Ewitt
**on [[Play Theory|http://webcat1.library.ubc.ca/vwebv/holdingsInfo?searchId=67969&recCount=10&recPointer=0&bibId=981498]] (Stephenson (1968)), cited by <<cite Case2008>>, <<cite Toms2000>>
**<<cite Toms2000>> on understanding and facilitating browsing: [[notes|Task Characterization: Meta Analysis]]
**<<cite Hutchins1995>> ch. 1: welcome aboard
*writing:
**[[we we explore|http://matthewbrehmer.net/2013/01/08/why-we-explore/]]
*LUNCH
!!09 January
*checking out [[IBM research opportunities|http://ibm-research.jobs/cambridge-ma/research-summer-intern-cambridge/33144214/job/]]
*checking out AT&T, Adobe, Autodesk, MSR
*meeting w/ TM: [[minutes|TM-13.01.09]]
*CV re-write, website updates
!!10 January
*email replay to JS re: Overview development & user study,
*email to Overview user MC on interview, follow-up usage questions
*email from CJ re: ~C-TOC follow-on study
*(dentist appt. 3:30pm)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
**<<cite Munzner2014>> draft textbook chapters, <<cite Munzner2009a>> (Vis. chapter)
*reading:
**[[The landscape of data analysis|http://simplystatistics.org/2013/01/10/the-landscape-of-data-analysis/]] by Jeff Leek, John Hopkins, also teaching a [[Coursera course on data analysis|https://www.coursera.org/course/dataanalysis]]
**Forbes article: [[R Is Not Enough For "Big Data"|http://www.forbes.com/sites/douglasmerrill/2012/05/01/r-is-not-enough-for-big-data/]] by Douglas Merrill
!!11 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
*reading:
**<<cite Roth2012>> on cartographic interaction primitives: [[notes|Task Characterization: Meta Analysis]]
**<<cite Norman1988>> on seven stages of action: [[notes|Task Characterization: Meta Analysis]]
*UDLS
!!References/Reading:
<<bibliography>>
!!14 January
*notes for <<cite Roth2012 bibliography:Bibliography>> on cartographic interaction primitives, <<cite Norman1988>> on actions/goals/tasks/intentions [[notes|Task Characterization: Meta Analysis]]
*reading:
**<<cite Roth2012b>> on empirically-derived cartographic interaction primitives: [[notes|Task Characterization: Meta Analysis]]
**<<cite Klein2006>>, <<cite Klein2006a>> on sensemaking: [[notes|Task Characterization: Meta Analysis]]
**<<cite Pike2009>> on the science of interaction [[notes|Task Characterization: Meta Analysis]]
**[[supermechanical: objects that connect us|http://supermechanical.com/objects.html]] via K. ~MacLean
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
*PM Mobify talk "From idea to reality, the twists & turns of building a business without startup funding" (5:30)
!!15 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
*[[Towards Personal Visual Analytics]] - guest presentation by Sheelagh Carpendale, University of Calgary (11:30-12:30)
*LUNCH
*Tuesday Tea
*reading:
**<<cite Lee2012a>> on beyond WIMP interaction in ~InfoVis: [[notes|Task Characterization: Meta Analysis]]
!!16 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
*meeting w/ JM+TM: [[minutes|JM-TM-13.01.16]]
*MUX meeting
*CTOC experiment code migration
!!17 January
*CTOC experiment code migration
*meeting w/ TM re: [[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk
**emailing MM re: DSM + Task Taxonomy
*meeting w/ RL re: business intelligence/VA at SAP
*DLS: Beth Mynatt (GT): Combining Computing, Design and Healthcare: Charting an Agenda for Personal Health Informatics (3:30 - 5:00)
!!18 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk preparation
*Demos & Lunch w/ Beth Mynatt
*TOCHI review request
*Overview
**emailing MK (recent Overview user) re: interview, follow-up w/ MC @ Tempero
*UDLS
*to read:
*to do: internship applications
**[[8 Insights About The Coming Era Of Interactive Design|http://www.fastcodesign.com/1671611/8-insights-about-the-coming-era-of-interactive-design]]
**<<cite Munzner2014>> text
!!References/Reading:
<<bibliography>>
!!21 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk for ~InfoVis group
*meeting w/ TM: debrief from pre-paper talk
*internship applications
**CV updates
*TOCHI review
**emailing editors re: deadline
!!22 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision
*TOCHI review
**printing / reading abstract
*Overview
**emailing MK (recent Overview user) re: interview next Wed (Jan 30)
**follow-up w/ MC @ Tempero
*CTOC
**email to CJ re: experiment logistics
**CT TOCHI journal paper Q
**experiment coding
*LUNCH - MN's context phone thingy
*Tuesday Tea
!!23 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision
*CTOC meeting re: experiment logistics
**experiment coding
**helping SH w/ ART ANOVA
*IMAGER social 12:30
!!24 January
*reading:
**[[Hands-On With the Next Generation Kinect: PrimeSense Capri|http://spectrum.ieee.org/automaton/robotics/robotics-hardware/handson-with-the-next-generation-kinect-primesense-capri/?utm_source=roboticsnews&utm_medium=email&utm_campaign=012213]] via KM
**[[Form Follows Function|http://fff.cmiscm.com/#!/main]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision
*Healthcare UX meetup
!!25 January
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision
*Overview related:
**response from user MC re: Overview workflow
**[[JMSC6041 Computational Journalism|http://jmsc.hku.hk/courses/jmsc6041spring2013/]] by JS, Journalism and Media Studies Center at the University of Hong Kong, Spring 2013
*reading:
**<<cite Buxton1986 bibliography:Bibliography>> on chunking and phrasing tasks: [[notes|Task Characterization: Meta Analysis]]
*UDLS
*to read:
**[[8 Insights About The Coming Era Of Interactive Design|http://www.fastcodesign.com/1671611/8-insights-about-the-coming-era-of-interactive-design]]
**<<cite Munzner2014>> text
!!References/Reading:
<<bibliography>>
!!28 January
*NG (Queens) HQP release of information request
*Overview:
**~P4K review dataset analysis
**response to user MC/Tempero re: Overview workflow
**follow-up with user MK re: Wed interview
*internship applications
**emailing HL re: Google UX position
**applied to MSR internships
**draft email to SD, MC @ MSR Vibe
*CTOC
**helping SH w/ ART ANOVA
**pilot ~BrainFreeeze2013 w/ PB: [[download|https://dl.dropbox.com/u/6397998/BrainFreeze2013.air]]
**emailing CT re: TOCHI submission - Friday 10am meeting
*reading:
**<<cite Fisher2012b bibliography:Bibliography>> on research directions w.r.t. HCI and big data analytics
**<<cite Kosara2013>> on storytelling and vis
*Checking out [[Ayasdi|http://www.ayasdi.com/]]
**requesting trial version?
>Our group's research has recently related to visualization and data analysis in the domain of computational journalism. I am interested in the user experience issues faced by professionals in this domain and I am familiarising myself with the workflows and tools available to journalists with varying degrees of technical expertise.
*checking out the [[Mitacs|http::/mitacs.ca/]] accelerate program
*Coursera [[Data analysis course|https://class.coursera.org/dataanalysis-001/class/index]]
**course intro, getting help, what is data?, representing data, representing data in R, editing R code
*Installed sublime text editor
!!29 January
*writing
*Coursera [[Data analysis course|https://class.coursera.org/dataanalysis-001/class/index]]
**simulation basics in R, types of data analysis questions, sources of data sets
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision
*internship applications
**response from HL: no vis-related UX research positions, only potential implementation-related positions; potential opportunities in the fall
**response from MS, RR re: MSR internship references
***to do: write reference material for RR
*CTOC
**fixing elementsMoved++ bug identified in pilot, [[new build|http://goo.gl/3TeOb]] sent to CJ, JM
*TOCHI review
*LUNCH
*Tuesday Tea
*reading:
**draft of IV journal version of <<cite Meyer2012>> for ~InfoVis reading group
!!30 January
*meeting w/ TM re: [[Task Taxonomy|Task Characterization: Meta Analysis]]: [[minutes|TM-13.01.30]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]] pre-paper talk revision 2.1
*internship applications
**response to HL:
**response from MS, RR re: MSR internship references
***to do: write reference material for RR
*setting up research chat w/ CPSC 554 team
!!31 January
*internship applications
**MSR email to VIBE (SD, MC, DF)
**writing reference material for RR
*~InfoVis group meeting: BELIV++ journal paper submission draft read (discussing <<cite Meyer2012>>)
*[[Task Taxonomy|Task Characterization: Meta Analysis]]
**pre-paper talk revision 2.1
!!01 February
*meeting w/ CT re: TOCHI submission [rescheduled to 02.05]
*reading:
**<<cite Card1997>> on The structure of the information visualization design space: [[notes|Task Characterization: Meta Analysis]]
**<<cite Spence2007>> ch. 2, 5 on interaction: [[notes|Task Characterization: Meta Analysis]]
**<<cite Wilkinson2005>> grammar of graphics: [[notes|Task Characterization: Meta Analysis]]
**<<cite Friel2001>> making sense of graphs: [[notes|Task Characterization: Meta Analysis]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]]
**pre-paper talk revision 2.1
**draft writing
*UDLS
!!References/Reading:
<<bibliography>>
!!04 February
*Overview:
**~P4K review dataset analysis
**MHK interview no-show (9am), interview (1pm)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*MUX guest talk on telecommuting by Tong Tang (Calgary)
*research chat w/ CPSC 554 team
*internship stuff
**reading MSR VIBE papers: <<cite Fisher2011 bibliography:Bibliography>>, <<cite Fisher2012a>>
*CTOC
**re-reading TOCHI in-the-wild reviews
*Coursera [[Data analysis course|https://class.coursera.org/dataanalysis-001/class/index]]
**structure of a data analysis pt. 1,2
!!05 February
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*internship stuff
**reading MSR VIBE papers
*CTOC
**meeting w/ CT re: TOCHI reviews, next steps
*LUNCH - SH interactions article: <<cite Churchill2013>>
!!05 February
*meeting w/ TM re MSR research / internship
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*internship stuff
**preparing for MSR VIBE interview
!!07 February
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
**reading: <<cite Bederson2003>> on theories in ~InfoVis: [[notes|Task Characterization: Meta Analysis]]
**reading: <<cite Tweedie1997>> on describing abstract externalizations: [[notes|Task Characterization: Meta Analysis]]
**library trip: checked out Stephenson's //Play Theory// (1968)
*internship stuff
**MSR VIBE interview
!!08 February
*reading:
**[[News headlines used to predict future events|http://www.bbc.co.uk/news/technology-21322203]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*~InfoVis group meeting: SI pre-paper talk
*UDLS
!!References/Reading:
<<bibliography>>
!!11 February
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*meeting w/ TM re: [[Task Taxonomy|Task Characterization: Meta Analysis]]
*Coursera [[Data analysis course|https://class.coursera.org/dataanalysis-001/class/index]]
**sources of data pt. 1,2
!!12 February
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*LUNCH
*Tuesday Tea
!!13 February
*(flight to FL)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*TOCHI review
!!14 February
*(in FL)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
*reading:
**<<cite Stephenson1967 bibliography:Bibliography>> on play theory: [[notes|Task Characterization: Meta Analysis]]
**<<cite Liu2010>> on mental models: [[notes|Task Characterization: Meta Analysis]]
!!15 February
*(in FL)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
!!18-22 February
*(in FL)
*CTOC study emails to CJ / minor code fixes
*registered for CHI
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
!!References/Reading:
<<bibliography>>
!!25 February
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft writing
**full draft complete, token to TM
*~InfoVis group meeting: JD research update on local node-link graph readability metrics, consideration lists for path following tasks, ongoing studies and implications for future predictive models
*CS Faculty candidate seminar: //Big Data Analytics w/ Parallel Jobs// Ganesh Ananthanarayanan (UC Berkeley ~AmpLab)
*CTOC emails re: logging, screen sharing, ~NASA-TLX, to CT, CJ
!!26 February
*config twedit on iOS
*emailing MSR's DF re: internship
*TOCHI review
*LUNCH - H&H 554m focus group
*Tuesday Tea
!!27 February
*passport renewal
*meeting w/ TM deferred - tbd
*applied for Autodesk internship
*applied to Adobe internship program
*IMAGER Social 12:30pm
*MUX: Gerhard Fischer (UC Boulder) on ~Meta-Design 2pm
*MUX talk prep
*TOCHI review
!!28 February
*MUX demos for GF, CW
*TOCHI review
!!01 March
*TOCHI review
*Seminar: //make it visible: applying cognitive systems engineering to intelligence analysis// William Wong (Middlesex U.)- 12pm DMP 301
*meeting w/ JM re: TOCHI review 1pm
!!04 March
*reading MB+TM paper draft discussion
*~InfoVis group meeting: MB+TM paper draft discussion
**debrief
*MUX talk prep
*TOCHI review
!!05 March
*(working from home - slightly ill)
*reading:
**[[A Pie in the Face for Information Visualization Research|http://www.perceptualedge.com/blog/?p=1492]] by Stephen Few, Perceptual Edge
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*TOCHI review submitting
*MUX talk prep
!!06 March
*meeting w/ TM re: [[Task Taxonomy|Task Characterization: Meta Analysis]] paper draft
*MUX: SN, MB
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*[[@hxdvancouver|http://www.meetup.com/HXD-Vancouver/events/105318782/?a=co1.3_grp&rv=co1.3]] meetup - talk on service design by [[@tamsina|https://twitter.com/TAMSINA]]
**[[Service Design Network|http://www.service-design-network.org/]]
**[[This is Service Design Thinking|http://thisisservicedesignthinking.com/]]
**[[John Seddon|http://en.wikipedia.org/wiki/John_Seddon]] (British occupational psychologist): //"If you manage for costs, costs go up, if you manage for value, costs go down"//
!!07 March
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*MUX talk: P. Baudisch (Hasso Plattner Inst.)
*PWIAS DLS: Colin Ware: [[The Process of Visual Thinking]]
!!08 March
*MUX demos for PB
*~InfoVis group meeting: JF pre-paper talk
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*UDLS
!!11 March
*~InfoVis group meeting: CW on visual thinking design patterns
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
!!12 March
*LUNCH
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
!!13 March
*(working from home)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft v2 complete - token to TM
!!14 March
*(climbing)
*booking CHI 2013 itinerary
*reading:
*[[If You Wear Google巠Glasses You Are An Asshole|http://gawker.com/5990395/if-you-wear-googles-new-glasses-you-are-an-asshole]] - Adrian Chen, Gawker
*FLS: A. Schaffer on perception and artist's sletches
!!15 March
*MUX demos for CS grad admits
*meeting w/ TM re: [[Task Taxonomy|Task Characterization: Meta Analysis]] draft
*TVCG abstract submitted
*UDLS
!!18 March
*reading SI VAST draft
*~InfoVis group meeting: discussing SI ~MoDisco VAST paper draft
*applying for internships: IBM, salesforce, facebook
*filing 2012 taxes
!!19 March
*reading <<cite Younos2012 bibliography:Bibliography>> for LUNCH
*LUNCH - discussing <<cite Younos2012>>
*checking out [[Wolfgang Alpha for FB profiles|http://www.wolframalpha.com/input/?i=my%20facebook#]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft token >> MB, draft editing
!!20 March
*Overview think-aloud usability studies (email from JS):
>''JS'': //"I want to see what naive users do on their first interaction with the system. The methodology I'm imagining is, put them in front of the site and tell them to "try out Overview. If you have a document set, upload it. Otherwise try one of the examples." Ask them to think aloud in the usual way, but say nothing. Do a screen recording."//
*joined [[Hacks/Hackers Vancouver|http://www.meetup.com/HacksHackersVancouver/]] meetup group
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft to JM for review
*[[Information is Beautiful on how we die|http://www.guardian.co.uk/news/datablog/2013/mar/18/information-beautiful-how-we-die?CMP=twt_gu]] - Gaurdian vis.
*MUX forum - JL on two-handed input
*RPE forum
!!21 March
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
*~InfoVis abstracts due (submitted, TM to confirm keyword choices)
*CTOC log file explaining in email
*CTOC log file analysis
*meeting w/ TM 2pm
!!22 March
*UDLS
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing - sent to JD, MM for feedback
!!References/Reading:
<<bibliography>>
!!25 March
*reading MS ~InfoVis draft
*~InfoVis group meeting: discussing MS ~InfoVis draft
*CTOC log file analysis
*meeting w/ JD to discuss [[Task Taxonomy|Task Characterization: Meta Analysis]] draft
!!26 March
*reading [[No to NoUI|http://www.elasticspace.com/2013/03/no-to-no-ui]] by Timo Arnall, elasticspace.com for LUNCH
*~TiddlyWiki, Firefox updates
*LUNCH - discussing [[No to NoUI|http://www.elasticspace.com/2013/03/no-to-no-ui]]
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing, feedback from MM
*(jamming w/ MH, OS)
*CTOC log file analysis - email to EC
!!27 March
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing, meeting w/ TM to discuss
*Imager pizza social
!!28 March
*reading JF ~InfoVis draft
*~InfoVis group meeting: discussing JF ~InfoVis draft
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing
!!29 March
*(Good Friday)
*[[Task Taxonomy|Task Characterization: Meta Analysis]] draft editing + submission
!!01 April
*(Easter Monday)
!!02 April
*writing
*reading:
**[[The Next Big UI Idea: Gadgets That Adapt To Your Skill|http://www.fastcodesign.com/node/1672044]] by Philip Battin, ~FastCo for LUNCH
**[[The top 20 data visualisation tools|http://www.netmagazine.com/features/top-20-data-visualisation-tools]] netmagazine via [[@EdwardTufte|http://twitter.com/EdwardTufte]]
*LUNCH - discussing [[The Next Big UI Idea: Gadgets That Adapt To Your Skill|http://www.fastcodesign.com/node/1672044]]
*CTOC log file analysis
!!03 April
*CTOC log file analysis
*Overview usability testing email drafts
*meeting w/ TM: [[minutes|TM-13.04.03]]
*MUX meeting: DT CHI practice talk
*meeting w/ JM re: CTOC log file analysis
*HXD meetup: HXD in developing nations - 7pm @ the Sin Bin
!!04 April
*reading:
**<<cite Holtzblatt1993 bibliography:Bibliography>> on [[Contextual Inquiry]]
*Overview:
**emailing JS re: fly-on-the-wall [[Think Aloud Protocol]] vs. [[Contextual Inquiry]]
*~InfoVis paper selection for Monday meeting:
**<<cite Dunne2013>> on node-link glyphs
*Annual progress reports:
**[[CS department|https://www.cs.ubc.ca/students/grad/policies/annual-phd]] (end of April)
**[[NSERC|https://www.grad.ubc.ca/forms/annual-progress-report-fellowship-holders]] (June 1)
*(climbing)
*updated OS X to 10.8
!!05 April
*Annual progress reports
*reading <<cite Dunne2013>> on node-link glyphs
*reading others' thesis proposals - downloaded template
*Overview contextual inquiry pitch emails - emailed KR (Vancouver Sun, UBC ~J-School), EU (Seattle Times), KM (UBC Vis. Cog lab)
**response from EU
*UDLS (did not attend)
!!References/Reading:
<<bibliography>>
!!08 April
*~InfoVis group discussion re: <<cite Dunne2013 bibliography:Bibliography>> on node-link glyphs
*Overview contextual inquiry pitch emails
**initiated conversation w/ [[CP @ Seattle Times|http://www.linkedin.com/pub/cheryl-phillips/8/b84/139]]
**Hacks/Hackers pitch
**response from JS re: contextual inquiry vs. fly-on-the-wall
*reading:
**<<cite Ulwick2002>> for LUNCH
**[[How ICIJ⯪ect Team Analyzed the Offshore Files|http://www.icij.org/offshore/how-icijs-project-team-analyzed-offshore-files]]
*IEEE ~InfoVis '13 paper reviews x2
*CHI '13 abstract scraping for Overview analysis
!!09 April
*reading:
**[[Django|https://www.djangoproject.com/]] (python web framework for journalists)
**[[PandaProject|http://pandaproject.net/]] for secure storage and searching newsroom data (a data library)
*Overview:
**continued conversation w/ [[CP (data enterprise editor) @ Seattle Times|http://www.linkedin.com/pub/cheryl-phillips/8/b84/139]]
**ICIJ offshore banking document dump - watching [[CBC exposԮ Merchant|http://www.cbc.ca/player/News/TV+Shows/The+National/ID/2369731413/]] by [[F. Zalac|http://www.icij.org/journalists/frederic-zalac]] (CBC, ICIJ - based out of Vancouver), [[H. Cashore|https://twitter.com/HarveyCashore]]
***searching ~LinkedIn for journalism contacts (e.g. Globe & Mail's data / vis journalist [[S. A. Thompson|http://www.stuartathompson.com/]] - based out of Toronto
*LUNCH - discussing <<cite Ulwick2002>> on new product development
*meeting w/ prospective student
!!10 April
*Overview:
**email to [[F. Zalac|http://www.icij.org/journalists/frederic-zalac]] (CBC, ICIJ - based out of Vancouver)
*meeting w/ TM: [[minutes|TM-13.04.10]] - deferred to 13.04.15
*lunch w/ prospective grads
*MUX meeting: talk from visiting student Antoine Ponsard
*CTOC log file analysis
!!11 April
*CTOC log file analysis
!!12 April
*CTOC log file analysis
*UDLS on tax havens and the #offshoreleaks
!!References/Reading:
<<bibliography>>
!!08 April
*reading + ~InfoVis group discussion re: <<cite Lin2013 bibliography:Bibliography>> on colour theme modelling
*reading:
**[[Who will hire all the PhDs? Not Canadaersities|http://www.theglobeandmail.com/news/national/education/who-will-hire-all-the-phds-not-canadas-universities/article10976412/]] - Melonie Fullick, Globe & Mail
**[[Academia's indentured servants|http://www.aljazeera.com/indepth/opinion/2013/04/20134119156459616.html]] - Sarah Kendzior, Al Jazeera
**[[The disposable academic - Why doing a PhD is often a waste of time|http://www.economist.com/node/17723223/comments?page=1#sort-comments]] - the Economist
**[[On Asking Questions and Academic Love|http://blogs.cornell.edu/danco/2013/02/18/on-asking-questions-and-academic-love/]] by Dan Cosley, Cornell (for LUNCH)
**<<cite Churchill2013>> on HCI curriculum (interactions article)
*reading thesis proposals (HL, DA, SI)
*meeting w/ TM: [[minutes|TM-13.04.10]] (deferred from 13.04.10)
!!09 April
*preparations for travel
*email contact w/ KT, [[Pulse Energy|http://ch.tbe.taleo.net/CH05/ats/careers/requisition.jsp?org=PULSEENERGY&cws=1&rid=67]]
*LUNCH - discussing [[On Asking Questions and Academic Love|http://blogs.cornell.edu/danco/2013/02/18/on-asking-questions-and-academic-love/]]
*CTOC video coding, email to EFC, MS, JM
!!10 April
*(departing for France)
!!11 April
*(vacation, Lyon)
!!12 April
*(vacation, Lyon)
!!References/Reading:
<<bibliography>>
!!To read:
*[[Tableau Software's Pat Hanrahan on "What Is a Data Scientist?"|http://www.forbes.com/sites/danwoods/2011/11/30/tableau-softwares-pat-hanrahan-on-what-is-a-data-scientist/]]
*[[8 Insights About The Coming Era Of Interactive Design|http://www.fastcodesign.com/1671611/8-insights-about-the-coming-era-of-interactive-design]] - via ~FastCo
*[[Reuters' Connected China|http://connectedchina.reuters.com/]]
!!06 May
*CHI 2013 reimbursal
*email catchup
*adding to-read items to [[References: CHI2013]]
*~InfoVis reviews
*DRITW review analysis
!!07 May
*LUNCH 12-1
*CTOC Log File meeting
*~InfoVis reviews
*DRITW review analysis
*Pulse Energy meeting prep
!!08 May
*meeting w/ TM, MS, SI re: DRITW TVCG reviews 9am via Skype
*meeting w/ TM, JM: minutes 12pm
*~InfoVis reviews
*Pulse Energy meeting prep
*meeting w/ KT @ Pulse Energy 3pm (downtown Seymour Street)
*H. Wickham speaks at Vancouver R user group meetup 7pm @ ~HootSuite 5 (East 8th Ave): [[Packages R Easy|http://bit.ly/pkgsrez]]
!!09 May
*updating R, ~RStudio
*~InfoVis reviews (due)
*researching Internship opportunities
*Pulse Energy follow-up
*M.Sc presentation: //~User-Centric Data Warehousing and Schema Understanding// (MS) 2pm ICICS 238
*H. Wickham seminar [[Bigvis: visualizing 100,000,000 observations in R]] 4pm ESB
!!10 May
*R review
*H. Wickham [[ggplot2 workshop|http://bit.ly/ggplot2ubc]] 1pm ESB
!!References/Reading:
<<bibliography>>
!!13 May
*(weekend) R/ggplot2 experimentation / data analysis
*Coursera browsing
*DRITW TVCG revisions
!!14 May
*(working from home)
*(BC provincial election)
*reading:
**<<cite Erickson2013 bibliography:Bibliography>> on domestic Electricity consumption portal (CHI '13): [[notes|Information Visualization Evaluation: Qualitative Methods]]
**[[FDIC and Bloomberg In Contact Over Data|http://online.wsj.com/article/SB10001424127887324216004578481534075601070.html]] - WSJ
**[[Gov't obtains wide AP phone records in probe|http://bigstory.ap.org/article/govt-obtains-wide-ap-phone-records-probe]] - AP
*meeting w/ TM 11h: [[minutes|TM-13.05.14]]
*DRITW TVCG revisions
*preparing talk slides for ~PhD annual progress report meeting
!!15 May
*(working from home)
*email from KT re: Pulse project ideas
**commentary and email to TM
**coordination re: meeting
*~InfoVis review discussion
*preparing talk slides for ~PhD annual progress report meeting
!!16 May
*~InfoVis review discussion
*reading:
**[[Laptop U: Has the future of college moved online?|http://www.newyorker.com/reporting/2013/05/20/130520fa_fact_heller]] by Nathan Heller, New Yorker
**[[Stories and Data: An Extraordinary Collaboration|http://www.perceptualedge.com/blog/?p=1632]] by Stephen Few, Perceptual Edge
*[[R Shiny tutorial|http://rstudio.github.io/shiny/tutorial/]]
!!17 May
*built R //Shiny ~P4K Album Review Explorer// with ggplot2, [[Shiny|http://www.rstudio.com/shiny/]] (to deploy as a [[gist|https://gist.github.com/]])
*wrote R ~earthquakeExplorer with ggplot2, Lat/Lon earthquake data
*resources:
**[[Shiny example: Diamonds Explorer|https://gist.github.com/jcheng5/3239667]]
**[[Height and weight of schoolchildren|http://glimmer.rstudio.com/winston/heightweight/]] ([[source|https://gist.github.com/wch/4034323]])
**[[Legends (ggplot2)|http://www.cookbook-r.com/Graphs/Legends_(ggplot2)/#modifying-the-text-of-legend-titles-and-labels]]
!!References/Reading:
<<bibliography>>
!!20 May
*(Victoria Day holiday)
!!21 May
*got ~RStudio service instance
*refining R //Shiny ~P4K Album Review Explorer// with ggplot2, [[Shiny|http://www.rstudio.com/shiny/]] (to deploy as a ~RStudio Server app)
*LUNCH: JD GI practice talk: //It's Alive: Exploring the Design Space of a Gestural Phone//
*preparing talk slides for ~PhD annual progress report meeting
*gathering [[References: Pulse]] ([[bibtex|Bibliography-Pulse]])
!!22 May
*refining R //Shiny ~P4K Album Review Explorer// with ggplot2, [[Shiny|http://www.rstudio.com/shiny/]] (to deploy as a ~RStudio Server app)
*preparing talk slides for ~PhD annual progress report meeting
*reading HS CSCW draft
*IMAGER pizza social
**HCI - SN: //How do reviewing motivations differ across reviewers?//
**Graphics - CM: //Dynamic Element Textures//
*MUX: HS CSCW draft review
*meeting w/ TM re: Pulse project ideas, annual progress report - minutes
!!23 May
*supervisory committee meeting 11-12 in x530
*meeting w/ TM, JM re: supervisory structure - minutes
*refining R //Shiny ~P4K Album Review Explorer// with ggplot2, [[Shiny|http://www.rstudio.com/shiny/]] (to deploy as a ~RStudio Server app)
!!24 May
*(working from home)
*released v1 of [[ShinyFork|http://spark.rstudio.com/mattbrehmer/ShinyFork/]]
*producing Pulse Energy meeting slides
*browsing [[timevis.net|http://www.timeviz.net/]], reading <<cite Aigner2011a bibliography:Bibliography-Pulse>> //Visualization of ~Time-Oriented Data// text
**ch. 1: Introduction
!!References/Reading:
<<bibliography>>
!!27 May
*(working from home)
*producing Pulse Energy meeting slides
*reading <<cite Aigner2011a bibliography:Bibliography>> //Visualization of ~Time-Oriented Data// text ([[timevis.net|http://www.timeviz.net/]])
**ch 3: time and data: [[notes on design aspects of time-oriented data|Information Visualization: Techniques]]
**ch 4: design: //what / why / how:// //why// overlaps w/ task taxonomy work: [[notes|Task Characterization: Meta Analysis]]
**ch. 5: interaction [[notes|Information Visualization: Techniques]]
*adding to [[References: Pulse]]
!!28 May
*browsing statcan.gc.ca, data.gc.ca
*reading <<cite Aigner2011a>> //Visualization of ~Time-Oriented Data// text ([[timevis.net|http://www.timeviz.net/]])
**ch 6: analytics: [[notes|Task Characterization: Meta Analysis]] (also overlaps w/ task taxonomy work)
*Seminar: //Biosignals and Interfaces// Tanja Schulz, Research Professors, Language Technologies Institute, CMU; Full Professor, Dept of Informatics, Karlsruhe Institute of Technology (KIT) (11am x836)
*LUNCH 12pm x860
*meeting w/ TM + SI 1:30 re: Overview design + case studies CHI paper
*automatic Skype premium account renewal
!!29 May
*reading <<cite Aigner2011a>> //Visualization of ~Time-Oriented Data// text ([[timevis.net|http://www.timeviz.net/]])
**ch 7: survey of visualization techniques: [[notes|Information Visualization: Techniques]]
**ch 8: conclusion and research challenges: [[notes|Information Visualization: Techniques]]
**ch 2: classic ways of showing time and time-oriented data
*time-oriented visualization reference gathering: [[References: Pulse]]
*refining Pulse Energy meeting slides
*reading <<cite Dork2012>> on information strolling, , <<cite Dork2011>> on the "information flaneur"
!!30 May
*meeting w/ Pulse (KT, TM, MB): minutes
*Seminar: //Recent Developments in Deep Learning//: Geoff Hinton (U of. T) at UBC (DMP 110 3:30pm)
!!31 May
*sent [[slides|http://goo.gl/R4e10]] to [[Pulse|http://www.pulseenergy.com/pulse-platform/pulse-energy-manager]], to iterate w/ TM on Mitacs application
**[[application docs|http://www.mitacs.ca/accelerate/apply-now/mitacs-accelerate]]
**[[UBC leave policy|https://www.grad.ubc.ca/current-students/scholarships-awards-funding/award-holders-guide]]
**[[NSERC leave policy|http://www.nserc-crsng.gc.ca/Students-Etudiants/Guides-Guides/PGSCGSRegs-ESESCRegs_eng.asp#interruption]]
*reimbursement claim for Skype Premium subscription sent to LS
*[[Scraperwiki|https://scraperwiki.com/profiles/mbrehmer/]] hacking
*Overview CHI paper brainstorming / pre-paper talk preparation
**revising [[References: Overview]]
**reading <<cite Liu2013>> on evaluating Newdle
!!References/Reading:
<<bibliography>>
!!03 June
*lab tour + lunch for Uta Hinrichs
*talk: Uta Hinrichs: //Open-Ended Explorations in Exhibition Spaces: A Case for Information Visualization and Large Direct-Touch Displays// (~PhD dissertation)
*~InfoVis summary review follow-up request from program chairs
*[[Mitacs|http://www.mitacs.ca/accelerate/program-guide]] application draft writing: [[getting started|http://www.mitacs.ca/accelerate/apply-now/interns-getting-started]]
*meeting w/ Uta Hinrichs
!!04 June
*[[Mitacs|http://www.mitacs.ca/accelerate/program-guide]] application draft writing
*LUNCH meeting 12pm
*CTOC TOCHI paper update email to CT, JM
!!05 June
*CTOC
**TOCHI paper update email to CT, JM
**~SproutCore follow-up email to CJ
*[[Mitacs|http://www.mitacs.ca/accelerate/program-guide]] application draft writing
*MUX meeting: Sharifa Alghowinem (Australian National University)
!!06 June
*~InfoVis review received: conditional acceptance!
**reading reviews
*[[Mitacs|http://www.mitacs.ca/accelerate/program-guide]] application draft writing; sent draft to TM
*CS dept. retreat 1-4pm
!!07 June
*~InfoVis revision notes
*[[ShinyFork|http://spark.rstudio.com/mattbrehmer/ShinyFork/]] tweaking + review text
*Pulse
**iterating on Mitacs application
**exploring Pulse Energy Manager tool
**Pulse @ ~FortisBC meeting? (permission to join not granted)
!!References/Reading:
<<bibliography>>
!!10 June
*~InfoVis revisions
**reading: <<cite Lammarsch2012 bibliography:Bibliography>>, <<cite Raskin2000>>, <<cite Andrienko2006>>, <<cite Aigner2011a>>: [[notes|Task Characterization: Meta Analysis]]
!!11 June
*meeting w/ TM re: ~InfoVis revisions and Mitacs application: [[minutes|TM-13.06.11]]
*LUNCH meeting 12pm
*~InfoVis revisions
!!12 June
*~InfoVis revisions
!!13 June
*~InfoVis revisions
**reading: <<cite Dork2011>>: the information flaneur: [[notes|Task Characterization: Meta Analysis]]
**reading: [[Typology versus taxonomy|http://www.island94.org/2010/06/typology-versus-taxonomy/]] by Ben Sheldon
**[[Taxonomy or typology? Theorising classifications of plants and animals in archaeology|http://www.academia.edu/1661275/Taxonomy_or_typology_Theorising_classifications_of_plants_and_animals_in_archaeology]] by David Orton
*CTOC TOCHI meeting w/ JM, CT 9:30
*CSGSA lunch
!!14 June
*~InfoVis revisions
*Pulse Mitacs proposal sent to KT, TM
!!15 June (yes, a Saturday)
*~InfoVis revisions, sent token to TM
!!References/Reading:
<<bibliography>>
!!17 June
*(day off, worked Saturday June 15)
!!18 June
*reading: [[Simplicity Is Highly Overrated|http://www.jnd.org/dn.mss/simplicity_is_highly.html]] by Don Norman (for LUNCH)
*~EuroVis paper roundup
*LUNCH meeting 12pm
*~InfoVis revisions: token TM -> MB, edits resume, cover letter
**reading: <<cite Bailey1994 bibliography:Bibliography>> and <<cite Smith2002>> on [[typologies vs. taxonomies|http://books.google.ca/books?id=1TaYulGjhLYC&lpg=PP1&pg=PA2#v=onepage&q&f=false]]
!!19 June
*MUX: CHI / GI / GRAND review session
*~InfoVis revisions + cover letter
**reading: <<cite Vicente1999>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
!!20 June
*(birthday)
*~InfoVis revisions + cover letter
**reading: <<cite Vicente1999>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
!!21 June
*reading: <<cite Vicente1999>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
*R text tools experimentation
*Talk: //Eliciting Informal Specifications from Scientific Modelers for Evaluation and Debugging// by Chris Bogart, Oregon State University (1:30, x836)
!!References:
<<bibliography>>
!!24 June
*Pulse Mitacs proposal comments from KT - editing proposal
*reading [[The four points of the HCI research compass|http://dl.acm.org/citation.cfm?doid=2451856.2451866]] by �Obrenovi.) interactions 20, 3, p.34-37
*~InfoVis revisions + cover letter (token MB -> TM)
**reading: <<cite Vicente1999 bibliography:Bibliography>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
!!25 June
*meeting w/ TM: [[minutes|TM-13.06.25]]
*~InfoVis revisions + cover letter edits
*Pulse Mitacs proposal editing
*LUNCH meeting 12pm, discussing [[The four points of the HCI research compass|http://dl.acm.org/citation.cfm?doid=2451856.2451866]]
!!26 June
*~InfoVis revisions (tweaking)
**reading: <<cite Vicente1999 bibliography:Bibliography>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
*MUX: SH ~MSc presentation
*Pulse Mitacs proposal - reply to KT
*[[ShinyFork|http://spark.rstudio.com/mattbrehmer/ShinyFork/]] updates
*Meetup: [[Analyzing User Behavior at Plenty of Fish: Data Science in the Wild|http://meetu.ps/1C1HbH]]
!!27 June
*~InfoVis cover letter revisions
**reading: <<cite Vicente1999 bibliography:Bibliography>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]]
**submitted final revisions
!!28 June
*writing
*notes on [[Cognitive Work Analysis]]
*Pulse Mitacs iteration / preparation for submission, sent for feedback to Mitacs' Sang Mah (Business Dev. Director for BC)
**reply to KT re: starting user data collection
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: ch. 2 - 4: Introducing D3, Technology Fundamentals, Setup
*exploring and making notes on Pulse Energy Manager
*reading:
**[[How to Report an F Statistic|http://www.yorku.ca/mack/RN-HowToReportAnFStatistic.html]] by I. S. Mackenzie, York U
*reply to MS' typology/taxonomy Q&A
!!References:
<<bibliography>>
!!01 July
*(Canada Day holiday)
!!02 July
*meeting w/ TM: [[minutes|TM-13.07.02]]
*Pulse
**exploring and making [[notes on Pulse Energy Manager - interface analysis|Pulse Energy Manager Notes]]
**emailing LZ, UBC energy manager re: contextual inquiry request
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: Data (ch. 5)
*reading <<cite Tilley2008 bibliography:Bibliography>> on research ethics for MUX
*meetup: [[Two Sessions: Telling Lies With Maps and Visual Analytics in Video Game Design|http://meetu.ps/1Nk56H]] - Vancouver Data Visualization - 6:30 pm Dell Inc.
!!03 July
*Pulse
**Mitacs minor revisions from SM: budget, timelime, interaction %, [[ethics|http://www.rise.ubc.ca/]], [[UBC ORS|http://www.ors.ubc.ca/contents/about]]
**exploring and making notes on Pulse Energy Manager - interface analysis
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: Drawing with Data (ch. 6)
*MUX meeting 2pm discussing <<cite Tilley2008>> on research ethics
!!04 July
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: Scales, Time Scales (ch. 7)
*[[Overview interview analysis - MK|Overview - MK interview transcript/notes]]
*reading DRITW draft and reviews
*email from JP: ~PhD Course requirement completed and confirmed; Thesis proposal must be completed by Aug 2014
!!05 June
*DRITW
**reading DRITW draft and reviews
**meeting w/ MS, TM, SI 9am
*[[Overview interview analysis - MK|Overview - MK interview transcript/notes]]
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: Axes (ch. 8), Updates, Transitions, and Motion (ch. 9)
*[[github vis libraries|https://github.com/bebraw/jswiki/wiki/Visualization-libraries]]
!!References:
<<bibliography>>
!!08 July
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray: Interactivity (ch. 10)
*re-read CTOC TOCHI submission
*exploring and making [[notes on Pulse Energy Manager - interface analysis|Pulse Energy Manager Notes]]
!!09 July
*[[update to TM|TM-13.07.09]]
*LUNCH meeting
*Pulse
**exploring and making [[notes on Pulse Energy Manager - interface analysis|Pulse Energy Manager Notes]]
**meeting w/ LZ, UBC energy manager re: contextual inquiry - 1pm
**note taking and revisions, skimming email attachments from LZ
**setting up Pulse SVN
*re-read CTOC TOCHI submission
!!10 July
*re-read CTOC TOCHI reviews
*cs.ubc account cleanup, mail cleanup, SVN dissertation setup, SVN client upgrade 1.6 > 1.7
*Pulse
**[[notes for LZ meeting|Pulse-LZ-13.07.09]]
**follow up emails to Mitacs BD, KT for energy manager contacts
*CTOC TOCHI paper meeting 2pm w/ JM + CT
!!11 July
*IEEE ~InfoVis final acceptances - accepted!
*email cleanup/organizing
*exploring/configuring [[HootSuite|http://www.hootsuite.com/]]
*Pulse: [[notes for LZ meeting|Pulse-LZ-13.07.09]]
*MN pilot study
!!12 June
*DRITW: TM token to MS, token to MB July 15
**reading cover letter
**reviewing diffs
*[[redesigning website|http://cs.ubc.ca/~brehmer/]], adding task typology paper
**and [[CHI paper project page|http://www.cs.ubc.ca/labs/imager/tr/2012/Interruptions/]]
*Pulse: [[notes for LZ meeting|Pulse-LZ-13.07.09]], reading attached documentation
!!To do:
*Overview CHI paper
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013 bibliography:Bibliography>> on evaluating Newdle
*CHI Doctoral Consortium abstract
!!References:
<<bibliography>>
!!15 July
*website updates - [[InfoVis '13 paper website|http://www.cs.ubc.ca/labs/imager/tr/2013/MultiLevelTaskTypology/]]
*reading:
**[[Why Canada is Failing at Tech|http://blog.hootsuite.com/why-canada-is-failing-at-tech/]] Ryan Holmes (~HootSuite), 07-11-13 in the Financial Post
**[[#Fminus: Why Universities are Failing at Teaching Social Media|http://blog.hootsuite.com/fminus-universities/]], Ryan Holmes (~HootSuite), 09-27-12 in Fortune
*Pulse:
**KT sent ~McGill energy manager contact
**[[notes for LZ meeting|Pulse-LZ-13.07.09]]
*DRITW: MS token, token to MB July 16
**reviewing diffs
*MS email re: design study survey paper and task typology
*participating in AK experiment 4pm
!!16 July
*meeting w/ TM: [[minutes|TM-13.07.16]]
*DRITW token; reading diffs, proofread pass
**emailing SI re: token, classification/regression issue, Kandogan/Ankerst refs.
*Pulse: emailing SM (Mitacs) re: ORS signature; emailing KT re: Pulse upcoming release, Mitacs signature (4pm Monday, Jul 22)
*no LUNCH meeting (JM @ [[MSR faculty summit|http://research.microsoft.com/en-us/events/fs2013/virtualfacultysummit.aspx]])
*Task Typology: camera-ready manuscript edits, figure and table edits, removing DRITW ref
!!17 July
*DRITW token; reading diffs, proofread pass
*Task Typology: new camera-ready cover letter
*MUX: LL on Mechanical Turk
*Mitacs ORS instructions from SM
!!18 July
*~InfoVis group meeting: JF practice talk - 3pm
**to do: group website edit access, party?
*[[Overview]] data analysis - MHK data set exploration
!!19 June
*CV, ~LinkedIn updates
*editing pass on CTOC TOCHI submission
*checking out [[vega|https://github.com/trifacta/vega]]
!!To do:
*[[Overview]] CHI paper
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013 bibliography:Bibliography>> on evaluating Newdle
*CHI Doctoral Consortium abstract
!!References:
<<bibliography>>
!!22 July
*R, Shiny, googleVis: [[tutorial|http://decastillo.github.io/googleVis_Tutorial/#75]], [[example|https://gist.github.com/decastillo/9443b6a9db6a1842aad9]]
*Pulse:
**Mitacs application revisions
**meeting w/ KT @ Pulse re: summer/fall release, Mitacs signatures
**given demo access to upcoming release
*TM sent KH monograph: [[Some Whys and Hows of Experiments in Human൴er Interaction|http://www.nowpublishers.com/articles/foundations-and-trends-in-humancomputer-interaction/HCI-043]]
!!23 July
*Overview pre-pre-paper draft
*LUNCH meeting
*Tuesday tea
*[[emailing RT (UBC PoliSci)|http://www.politics.ubc.ca/graduate-program/phd-profiles/richard-togman.html]]
*Pulse: emailing JC, energy manager @ ~McGill
*checking out [[rickshaw.js|http://code.shutterstock.com/rickshaw/]] - toolkit for creating interactive time series graphs
!!24 July
*Pulse:
**responding to JC, energy manager @ ~McGill, setting up Skype meeting 11am Monday July 29
**checking out D3 demos and related technology:
***[[crossfilter.js|https://github.com/square/crossfilter]] by square
***[[cubism.js|https://github.com/square/cubism]] by square
***[[F+C via brushing|http://bl.ocks.org/mbostock/1667367]], [[day/hour heatmap|http://bl.ocks.org/tjdecke/5558084]], [[calendar view|http://bl.ocks.org/mbostock/4063318]]
**KT sent UCB contact
*DRITW author bio / pictures to TM
*~InfoVis group meeting: JD CHI pre-paper talk
*MUX: JF M.Sc thesis: //Variant View: Visualizing Genetic Sequence Variants in their Gene Context//
*email from JS re: Overview tag loading
!!25 July
*Installing [[LaTeXTools|https://github.com/SublimeText/LaTeXTools]] and [[Package Control|http://wbond.net/sublime_packages/package_control/installation]] for Sublime Text 2, [[Skim PDF viewer|http://sourceforge.net/projects/skim-app/?source=dlp]]
*new [[about.me page|http://about.me/mattbrehmer]]
*Overview pre-pre-paper draft
*meeting w/ TM 2pm: [[minutes|TM 13.07.25]]
*uploaded Task Typology copyright form
!!26 June
*Overview pre-pre-paper draft
*BCCDC hosts [[Jer Thorp|http://about.me/jerthorp]] 11:30am
!!29 July
*Pulse
**[[meeting|Pulse-JC-13.07.29-MEB-13.08.03]] w/ [[JC|http://www.linkedin.com/pub/jerome-conraud/26/853/214]], energy manager @ [[McGill|http://www.mcgill.ca/facilities/utilities/dashboard]] via Skype + screencast recording: [[notes|Pulse-JC-13.07.29-MEB-13.08.03]]
**contacted SES consultants, UBC BMS, [[McGill|http://www.mcgill.ca/facilities/utilities/dashboard]] student intern w/ [[McGill Energy Project|http://mcgillenergyproject.wordpress.com/]], UCB* contacts (* available mid-August)
*checking out [[Bret Victor|http://worrydream.com/#]]'s website (via [[@EdwardTufte|http://twitter.com/EdwardTufte]])
*learning d3, [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], [[processing.js|http://processingjs.org/]]
*Task Typology: fixed "the the" typo
!!30 July
*learning d3 re: [[scatterplot focus and context|http://jsfiddle.net/PyvZ7/7/]]
*Overview pre-pre-paper talk
*Pulse: responses to SES, ~McGill energy project
*No LUNCH
*CTOC TOCHI revisions
!!31 July
*Fixed TVCG template mailing date as per editors' request
*~InfoVis group meeting 11am; next week
*LUNCH @ Point Grill 12pm
*Overview meeting w/ SI, TM 1-3pm
**to do: tweak dataset, send to TM, JS, pre-paper talk in 2 weeks
*updated Shiny server
*sent CTOC TOCHI revision to CT
!!01 August
*Task typology camera-ready revisions due (submitted)
**uploaded teaser image
*Pulse: checking out MEB's [[McGill Steam Predictions project|http://mcgill-steam.herokuapp.com/]] ([[github repo|https://github.com/aj0strow/mcgill-steam]]), using the [[Pulse API|http://developer.pulseenergy.com/]]
*DT design pilot
*Overview: tweaking pre-paper talk
*email from ~Hans-J㣨ulz re: ~InfoVis '13 task paper, reading his paper
*tweeted Task Typology paper
!!02 August
*Overview
**new story (Jul 27, 2013), the reporter:
>//I was able to make sure that all the emails with a certain keyword were in a particular node on the branching diagram that overview created. That way I knew I had them all and hadn't missed any in my scan. I could have used document cloud to just find the key words, but by using overview, i also had emails closely related to the keyword that didn't necessarily contain that keyword.//
**to interview this reporter between Aug 19-21
**a G&M reporter working with Overview currently
**responding to JS emails re: meeting times
**new [[Overview video|http://vimeo.com/71483614]], updates include full text search, pdf upload (soon)
>''JS''://Development continues. In the medium term it's mostly about workflow. (All of this project is mostly about workflow, it turns out. Algorithms are the easy bit.) //
**Overview study research proposal
**to watch: [[Text Analysis in Transparency 䡬k at Sunlight Labs|http://overview.ap.org/blog/2013/05/video-text-analysis-in-transparency/]] by J. Stray (Sunlight Labs, May 2013)
*reading Schulz ~InfoVis13 paper on tasks
*reading about [[nanocubes|http://www.nanocubes.net/]] and [[imMens|http://vis.stanford.edu/papers/immens]]
*Pulse: interview w/ MEB, student with the ~McGill energy project: [[notes|Pulse-JC-13.07.29-MEB-13.08.03]]
!!05 August
*(BC Day holiday)
!!06 August
*Pulse: meeting CG @ SES Consulting: [[minutes|Pulse-SES-CG-13.08.06]]
*reading Schulz ~InfoVis13 paper on tasks
*reading about [[nanocubes|http://www.nanocubes.net/]] and [[imMens|http://vis.stanford.edu/papers/immens]] for ~InfoVis group
*watching [[@jonathanstray|http://twitter.com/jonathanstray]] on [[CSPAN|http://www.c-span.org/Live-Video/C-SPAN/]] re: NSA surveillance
!!07 August
*reading [[bigvis|http://vita.had.co.nz/papers/bigvis.pdf]] for ~InfoVis group
*~InfoVis group meeting 11am discussing [[nanocubes|http://www.nanocubes.net/]], [[imMens|http://vis.stanford.edu/papers/immens]], and [[bigvis|http://vita.had.co.nz/papers/bigvis.pdf]]
*Overview: reading JS' usability notes, watching [[Text Analysis in Transparency 䡬k at Sunlight Labs|http://overview.ap.org/blog/2013/05/video-text-analysis-in-transparency/]]
*MUX meeting on research paper writing
*Overview pre-paper talk
*Task typology camera-ready edits to satisfy TVCG request re: spacing, author bullets
*watching [[satisfy the cat|http://www.youtube.com/watch?v=dln9xDsmCoY]]
*meeting w/ JS, SI, TM: [[minutes|TM-JS-SI-MB-13.08.07]]
!!08 August
*watching JS' [[NewsU|https://twitter.com/newsu]] webinar [[Document Mining with Overview: A Digital Tools Tutorial|http://www.newsu.org/digital-tools-overview]]
*meeting w/ TM: [[minutes|TM-13-08-08]]
*reading [[give up on the net?|http://buzzmachine.com/2013/08/08/give-up-on-the-net/]] by [[Jeff Jarvis|http://twitter.com/jeffjarvis]], Buzz Machine
*[[Overview not clustering the way you髥? Try ignoring words|http://overview.ap.org/blog/2013/08/overview-not-clustering-the-way-youd-like-try-ignoring-words/]] - Overview blog (08.08.13)
*Overview pre-paper talk
!!09 August
*Overview pre-paper talk
!!12 August
*booked travel Sept 19 - Oct 1
*Task typology: email from Bordeaux ~PhD student
*Pulse
**Mitacs application under review;
**UC Berkeley energy manager interview Aug 26-30
*Overview pre-paper talk
*reading [[Are you a data scientist?|http://www.perceptualedge.com/blog/?p=1719]] by Stephen Few, Perceptual Edge
*reading [[The history of the future: sci-fi movies and HCI|http://dl.acm.org/citation.cfm?id=2486240&CFID=353480467&CFTOKEN=58689376]] by Aaron Marcus in ACM interactions
!!13 August
*Overview pre-paper talk
**comparative analysis figure construction
**bibtex building
*no LUNCH
*Tuesday Tea
!!14 August
*~InfoVis group meeting 11am: MB Overview pre-paper talk
**pre-paper debrief
**Overview pre-paper overhaul
!!15 August
*Overview pre-paper overhaul
!!16 August
*reading:
**[[Nate Silver addresses assembled statisticians at this year's JSM|http://blog.revolutionanalytics.com/2013/08/nate-silver-jsm.html]] by J. Rickert
**[[Save your work 楠software engineers a career track|http://www.timeshighereducation.co.uk/news/save-your-work-give-software-engineers-a-career-track/2006431.article]] by Chirs Parr, Times Higer Education
**[[When nerds and words collide|http://ire.org/blog/ire-news/2013/08/15/when-nerds-and-words-collide/]] ([[PDF|https://s3.amazonaws.com/s3.documentcloud.org/documents/757701/nerds-and-words.pdf]]) 1999 short book on CAR by the Poytner Institute
*meeting w/ TM: minutes (pre-paper debrief)
**[[document mining defnition|https://www.dagstuhl.de/en/program/calendar/semhp/?semnr=11171]]
*Overview pre-paper overhaul
!!19 August
*Overview paper draft
**[[Chuang et al.'s Stanford dissertation browser|http://www-nlp.stanford.edu/projects/dissertations/browser.html]]
**[[J. Eisenstein (Georgia Tech)|http://www.cc.gatech.edu/~jeisenst/]]
**gathering tag cloud refs
**reading text visualization RW re: tasks
*Pulse:
**emailing UBC building operations
*R / updating ~ShinyFork / ~TF-IDF experimentation
!!20 August
*Overview paper draft
*LUNCH
*OS study
!!21 August
*Overview paper draft
*checking out [[JS' upcoming comp. journalism course @ Columbia|http://www.compjournalism.com/]]
!!22 August
*Overview paper draft
*Overview: interviewing Overview user (off-the-record), making [[notes|Overview interview notes 13.08.23]]
**[[NICAR-2013|https://www.ire.org/conferences/nicar-2013/]]
**[[IRE-2013|http://ire.org/conferences/ire-2013/]]
**[[IRE-2013-Overview|http://ire.org/events-and-training/event/21/964/]]
!!23 August
*Pulse: scheduling interview w/ UCB's energy manager KN
*Overview paper draft
!!26 August
*Overview paper draft
*Pulse to do: call/email UBC BOS Pulse user
*reading SH CHI paper draft
!!27 August
*Overview paper draft
*LUNCH: SH CHI paper draft
*Pulse:
**scheduling meeting w/ BE, UBC Operations
**Mitacs application approved with minor revisions
*Task Taxonomy: fast forward slides/video due Sept. 24
!!28 August
*Overview paper draft
*[[Global Marine Traffic visualization|http://www.marinetraffic.com/]]
*Pulse:
**reviewing [[Pulse-JC-13.07.29-MEB-13.08.03]], transcript/notes from recorded interview
**examining [[CG-SES|Pulse-SES-CG-13.08.06]]'s Excel analysis documents
**meeting w/ KN, Energy Manager at UCB
*CTOC - signing publication release form for CJ
!!29 August
*Overview paper draft
*meeting w/ TM: [[minutes|TM-13-08.29]]
!!30 August
*~InfoVis group meeting, TM GD practice talk
*Overview paper draft
!!02 September
*(labour day)
!!03 September
*reading JL CHI paper draft
*LUNCH: JL CHI paper draft
*Pulse:
**Mitacs proposal revisions
*Overview paper draft
!!04 September
*meeting w/ JP re: Mitacs Accelerate / NSERC + 4YF funding
*Pulse: pinging KT, NV (SES)
*Task Taxonomy:
**video preview: [[draft|http://goo.gl/b1uqob]]
**browsing accommodations in ATL
**renewed ACM membership
*Overview paper draft
!!05 September
*Pulse: meeting BE, UBC building operations: [[notes|Pulse-BE-13.09.05]]
*Overview paper draft
*meeting w/ TM: [[minutes|TM-13-09.05]]
!!06 September
*~InfoVis group meeting, SI ~PhD defense practice talk
*Overview paper draft
!!07 September
*Overview paper draft, token to TM
*[[Task Typology video preview|http://www.cs.ubc.ca/labs/imager/tr/2013/MultiLevelTaskTypology/brehmer_infovis13.mp4]]
!!09 September
*(in Chilliwack, worked Saturday 13.09.07 instead)
*Pulse: Mitacs cover letter
!!10 September
*reading LO CHI paper draft
*reading JD IV journal paper draft
*Pulse: submitted revisions and cover letter
*reading B. Shneiderman's comments re: DSM
*IEEE VIS FF slides
!!11 September
*reading JD IV journal paper draft
*HCI@UBC
*MUX: LO paper review / feedback
*~InfoVis group meeting: JD IV journal paper draft
!!12 September
*CHI Doctoral Consortium abstract: pre-paper slides
*meeting w/ TM: [[minutes|TM-13.09.12]]
*Task Typology; [[IEEE VIS papers announced|http://ieeevis.org/year/2013/paper-session/all/infovis]]
**contacting RE Roth re: papers
!!13 September
*Task Typology: response from RE Roth re: pre-print of his article
*Overview
**TM token to MB? review edits / comments
**skimming Ribarsky et al VAST paper pre-print on hierarchical topic visualization for document collections
*SI ~PhD defense
*Pulse: Mitacs Accelerate application approved; contacting FOGS/NSERC for interruption of award; [[NSERC guidelines|http://www.nserc-crsng.gc.ca/Students-Etudiants/Guides-Guides/PGSCGSRegs-ESESCRegs_eng.asp#interruption]]
*L. Wilkinson seminar: //~High-Dimensional Visual Analytics: Exploring structure using low-dimensional projections//
*CHI Doctoral Consortium abstract
*meeting w/ L. Wilkinson
**[[Mallet:MAchine Learning for LanguagE Toolkit|mallet.cs.umass.edu/about.php]]
**[[r4stats.com|http://r4stats.com/2013/05/14/beginning-of-the-end-v2/]]
!!16 September
*meeting w/ S. North
*Overview paper draft edits, sent to group for review
*Pulse: Mitacs funding details sent to TM
!!17 September
*Task Typology: IEEE VIS FF slides
*CHI Doctoral Consortium abstract
!!18 September
*floor wardens meeting 10am ICICS 146
*reading:
**[[Academy Fight Song|http://thebaffler.com/past/academy_fight_song]] by Thomas Frank, The Baffler
**[[Do Microsoft employees use Google at the office?|http://www.quora.com/Microsoft/Do-Microsoft-employees-use-Google-at-the-office?__pmsg__=+M1JVUkpVVnRLei1qczZVNW9tNW06YS5hcHAudmlldy5wbXNnLmFsbC5Mb2dnZWRJbkZyb21MaW5rOltbMjA4Mzk2MDddLCB7fV0*]] - Quora
*~InfoVis group meeting 11am: discussing Overview draft, VIS FF slides
**debrief w/ TM
*Pulse: meeting w/ NV, SES consulting 4pm
*beers with ~InfoVis group @ Portland Craft
!!19 September
*CHI Doctoral Consortium abstract
*Pulse: NV sent Excel notes
*CERC DM presentation, lunch
*(flight to Ottawa)
!!20 September
*(in Ottawa)
*CHI Doctoral Consortium abstract
*Overview: meeting w/ JS
**[[Using clustering to analyze the voting blocs in the UK House of Lords|http://www.compjournalism.com/?p=13]], Jonathan Stray, Columbia Computational Journalism course
**[[Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review|https://litigation-essentials.lexisnexis.com/webcd/app?action=DocumentDisplay&crawlid=1&doctype=cite&docid=17+Rich.+J.L.+%26+Tech.+11&srctype=smi&srcid=3B15&key=2f43e4b280fafa588e38edb02452824c]] by Grossman and Cormack (2011), Richmond Journal of Law and Technology
**[[An Investigation of Linguistic Features and Clustering Algorithms for Topical Document Clustering|http://dl.acm.org/citation.cfm?id=345582]] by Hatzivassiloglou et al. (2000, SIGIR)
!!30 September
*(in Ottawa)
*catching up on email after week away
*DRITW TVCG rejection
*Task Typology talk prep
**reading Task Typology paper, RE Roth paper, Schꠥt al. paper
*CHI Doctoral Consortium extended abstract edits
!!01 October
*(flight to Vancouver)
*Task Typology talk, fixed typos
*LUNCH
*CHI Doctoral Consortium extended abstract edits, submitted version
*checking out [[GA Tech's CitVis|http://www.cc.gatech.edu/gvu/ii/citevis/]]
*GRAC emails
!!02 October
*Task Typology talk prep
*~InfoVis group meeting: TM ~InfoVis panel talk practice
*MUX: PB + MN research updates
*Pulse: meeting w/ KT 4pm
!!03 October
*Task Typology talk prep
*meeting w/ TM: discuss Task Typology talk: [[minutes|TM-13.10.03]]
!!04 October
*MS practice talk
*DRITW meeting
*Task Typology talk prep
*CHI Doctoral Consortium extended abstract due - submitted
!!07 October
*Task Typology practice talk for TM
**debrief w/ TM
*JD M.Sc practice defense
*meeting w/ TM (continued debrief)
!!08 October
*Task Typology talk practice
*LUNCH
!!09 October
*HCI@UBC 12 at IKBLC
*Task Typology practice talk for ~InfoVis group
**debrief w/ TM
*Task Typology talk revisions
!!10 October
*Task Typology talk revisions, practice, record and send to MS
*JD M.Sc defense 146 at 3pm
!!11 October
*Task Typology talk practice, feedback from MS
*Pulse Intro
*CHI review request
!!To do:
*Overview
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013 bibliography:Bibliography>> on evaluating Newdle
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!21 October
*(at home, ill)
*cleaning and distributing [[IEEEVIS 2013 Notes]]
*GRAND reporting
*CHI review
!!22 October
*IEEEVIS reimbursements
*ICICS/CS safety checklist
*344 guest lecture scheduling
*website / CV updates
*Pulse requirements analysis
!!23 October
*updating OSX
*reading <<cite Gorg2013 bibliography:Bibliography>> TVCG on Jigsaw, <<cite Dou2013a>> VAST '13 on Hierarchical Topics
*~InfoVis group meeting cancelled
*IMAGER social
*meeting w/ TM, JS on Overview evaluation study
!!24 October
*Pulse meeting w/ Surrey school board
*Pulse requirements analysis
*[[HXD meetup: Designing a Career: From Clinician to App Developer|http://www.meetup.com/HXD-Vancouver/events/144625982/]]
!!25 October
*Overview: emailing JS re: http://bootstraptour.js.com
*ORS ethics renewal
*Pulse requirements analysis
*UDLS
!!To do:
*Overview paper revisions
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013>> on evaluating Newdle
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
**read: [[Stephen Few on "What makes a visualization memorable" (Borkin et al. Proc. InfoVis '13)|http://www.perceptualedge.com/blog/?p=1770]]
!!References:
<<bibliography>>
!!28 October
*read [[Stephen Few on "What makes a visualization memorable" (Borkin et al. Proc. InfoVis '13)|http://www.perceptualedge.com/blog/?p=1770]]
*(starting at Pulse)
**tour of office, meeting team
**setup laptop, installing software, configuring preferences and SVN, upgtading OSX
**team lunch to celebrate Pulse Check reports successful launch
**meeting w/ Paul to chat about R/Shiny, analytics HQ
**meeting w/ KT to discuss plan for next couple of weeks
**requirements analysis: transcribing KN (UCB) interview
!!29 October
*Pulse:
**met JC, scheduled meeting for next week to discuss energy manager analysis tasks and workflows.
**requirements analysis
**completed notes for JC, KN interviews; notes to do: NV (SES), MT (Surrey SB)
*meetup: [[Data Science, Machine Learning, Marketing, and the Customer Experience|http://www.meetup.com/DataScience/events/142278212/?a=cr1_grp&rv=cr1&_af_eid=142278212&_af=event]], lightning talks by Pulse Energy and J. Bryant (UBC Stats) @ HootSuite
!!30 October
*~InfoVis group meeting cancelled
*CPSC 344/544 guest lecture abstract / bio
*Overview: reading JS's experiment design draft
*meeting w/ TM, JS on Overview evaluation study: [[minutes|TM-JS-10.30.13]]
!!31 October
*Pulse:
**requirements analysis
**attended weekly dev. roundtable update meeting
!!01 November
*Pulse:
**weekly ~EnergyCheck product demo
**requirements analysis
**presentation to pulsars re: HCI / ~InfoVis research at UBC
**attended meeting re: Energy Manager for Ontario natural gas customers
!!02 November
*CHI review
*Overview: examining Jigsaw
!!To do:
*Overview paper revisions
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013 bibliography:Bibliography>> on evaluating Newdle
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!04 November
*Pulse
**requirements analysis
**project update
**[[meeting w/ JC and KT to discuss Energy Manager|Pulse-JC-KT-13.11.04]]
!!05 November
*Pulse:
**requirements analysis / task abstraction analysis
*Overview:
**emailing user SA re: consent to cite story in Overview paper
!!06 November
*meeting w/ JM and TM to discuss Overview student study
*meeting w/ TM: [[minutes|TM-13.11.06]]
!!07 November
*Pulse:
**weekly dev. roundtable update meeting
**task abstraction analysis
**consolidated energy manger interview data meeting w/ KT
***[[CEUS: California End-Use Survey|http://www.energy.ca.gov/ceus/]]
!!08 November
*Pulse:
**data abstraction analysis, reading Pulse API documentation, access of data via the Pulse Platform, playing with Arilou demo
**JL referral
!!09 November
*Overview:
**emailing JS re student study
**CPSC 344/544 lecture prep
!!10 November
*Overview:
**CPSC 344/544 lecture prep
!!11 November
*(Remembrance Day holiday)
*Overview paper revisions
!!12 November
*Pulse:
**data abstraction analysis
**exploration of ~MySQL test database
**analytics brainstorming lunch, analytics huddle
**[[SaveHeat heatmap satellite images of Calgrary| neighbourhoodhttp://www.saveheat.co/heat-scores.php?address=3657%20W27%20ave,%20vancouver]]
**project update
!!13 November
*Overview paper revisions
**CPSC 344/544 guest lecture prep
*HCI@UBC: //Designing for Everyday Design Practices// - Ron Wakkary (SFU SIAT)
*meeting w/ TM: [[minutes|TM-13.11.13]]
!!14 November
*CPSC 344/544 guest lecture
*Pulse:
**translating data and task abstractions back into requirements / domain language: high-mid-low
!!15 November
*Pulse:
**bi-weekly demo
**reading about [[point types|https://my.pulseenergy.com/help/reference/pointTypes]], [[quantities|Energy Quantity Resource]]
**meeting w/ KT to discuss data/task abstractions
**converting abstractions into design proposal (back to domain vocabulary)
**contacting JC (Pulse), JC (~McGill), KN (UBC)
*reading <<cite Gratzl2013 bibliography:Bibliography>> et al.'s //~LineUp// VIS '13 paper.
!!16 November - 17 November
*Overview paper revisions
!!To do:
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
**[[notes|Information Visualization Evaluation: Quantitative Methods]] for <<cite Liu2013>> on evaluating Newdle, <<cite Dou2013a>>, <<cite Gorg2013>>
***read <<cite Kang2012>>
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!18 November
*Pulse:
**continuing to read <<cite Gratzl2013 bibliography:Bibliography>> et al.'s //~LineUp// (VIS '13), playing with ~LineUp
**sketching / mockups for energy manager proposal
**meeting w/ Pulse's JC re: proposal
*Overview:
**access to SA documents
!!19 November
*Pulse:
**continuing to sketch / wrangle data
**reading <<cite Gratzl2013 bibliography:Bibliography>> et al.'s //~LineUp// (VIS '13), playing with ~LineUp
!!20 November
*Overview paper revisions, token to TM
*meeting w/ TM: [[minutes|TM-13.11.20]]
!!21 November
*Pulse:
**preparing sketches for meeting w/ JC (~McGill)
!!22 November
*Pulse:
**preparing sketches for meeting w/ JC (~McGill)
**meeting w/ JC (~McGill), notes sent to Pulse's KT, JM
!!23 November - 24 November
*Overview paper revisions, token to JS, emailing Dallas journalist re: time estimates
!!To do:
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!25 November
*Pulse:
**sketching / mockups for energy manager proposal (UCB)
**continuing to sketch / wrangle data
!!26 November
*Pulse:
**sketching / mockups for energy manager proposal (UCB)
**continuing to sketch / wrangle data
*meetup: [[Data Science:Exploratory Analysis, Machine Learning, Statistics, Prediction and Visualization|http://www.meetup.com/DataScience/events/149038502/]] (Tu) 13.11.26 18h30, @HootSuite
!!27 November
*Overview:
**paper revisions, token from JS to MB
**to do: email JS re: student study follow-up, email Jigsaw team re: scalability
*Imager social
*meeting w/ TM: [[minutes|TM-13.11.27]]
!!28 November
*Pulse:
!!29 November
*Pulse:
*Overview:
**~EuroVis abstract due
!!30 November - 01 December
*Overview paper revisions
!!To do:
**read [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
*learn [[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!02 December
*(@UBC in AM)
*GRAC meeting to review GATS procedure
*Pulse:
**sketching / mockups for energy manager proposal (UCB)
!!03 December
*Pulse:
**sketching / mockups for energy manager proposal (UCB)
**wrote a data wrangling script for loading Pulse client data into a format that [[LineUp|http://lineup.caleydo.org]] likes
**portfolio sandbox: updated filter UI, line plots
**meeting w/ KT
***web app deployment with [[nginx|http://nginx.org/en/]]
***[[highcharts.js|http://www.highcharts.com/]]; ask NG about other charting libraries / packages for time-series data
!!04 December
*(@UBC)
*Pulse:
**portfolio sandbox: adding library calls to plyr and data.table packages for standalone sourcing and shiny::runApp() calls, refactored global.R to boot from local CSV rather than Pulse API calls, attribute type coercion, added conditional filters to UI
*Overview
**reviewing TM revisions, JD feedback, token to SI
**to do: email JS re: student study follow-up, email Jigsaw team re: scalability
*MUX meeting: SN on pee-review anonymity and meta-reviewing
*meeting w/ TM: [[minutes|TM-13.12.04]]
!!05 December
*Pulse:
**portfolio sandbox: refactoring data munging for one-pass filtering and faceting
*Overview:
**final edits based on SI's comments
!!06 December
*Overview paper submission
*Pulse:
**[[ggplot bump chart|http://learnr.wordpress.com/2009/05/06/ggplot2-bump-chart/]] experimentation
**adding bump chart to portfolio sandbox, added controls for aggregation and quantities, legible date intervals, dynamic chart height function
**meeting/demo w/ DH
**meeting w/ KN (UCB energy analyst), annotating meeting notes, reviewing [[2012 energy benchmarking report from San Francisco Water Power Sewer|http://sfwater.org/modules/showdocument.aspx?documentid=4139]]
**Pulse Christmas party
!!To do:
*[[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!09 December
*Pulse:
*summarizing meeting notes from KN / UCB interview, sending to KT
**portfolio sandbox: added [[ggplot heatmap|http://learnr.wordpress.com/2010/01/26/ggplot2-quick-heatmap-plotting/]] and boxplot charts, data munging code refactoring, attempting to parallelize API calls, removing deprecated ggplot references, fixed faceting by year
!!10 December
*Pulse:
**portfolio sandbox: data table and munging code refactoring, factored out data functions, consolidated bump plot code and removed text labels from all intervals, experimenting with [[horizon charts for time-series data in ggplot|http://timelyportfolio.blogspot.ca/2012/08/horizon-on-ggplot2.html]], added additional normalization attributes
***to do: add normalization calls to box plot
**meeting w/ KT
***KT to arrange meetings with BG, US utilico energy portfolio analysts
***a summary of what was discussed yesterday w/ KT regarding next steps in terms of exploring and prototyping visualizations for portfolio-level analysis. These items are sorted from big-picture ill-defined design problems to more concrete and well-defined design ideas
****''major'': understanding the workflow from portfolio-level analysis to portfolio-detail analysis, (also: gathering more user data by speaking to utility company users)
****''major'': design implications for tagging spaces (awaiting space-tag lists from JC (~McGill) and KN (UCB))
****''major'': indicators of missing data (# records), standard deviation, and ranges in aggregated data (as you saw, boxplots don't scale)
****''mid'': approaches to multi-attribute weighted rankings (졠[[LineUp|lineup.caleydo.org]]), flexibility vs. defined meaningful weightings
****''mid'': coordinated bar and bump charts (also 졠[[LineUp|lineup.caleydo.org]]; Ikely hitting a wall with Rඩsualization package options here, given that this is a fairly novel visual encoding combination; I may need to prototype in a different language environment to explore designs in this area).
****''minor'': HDD/CDD normalization: determine which spaces in a portfolio are correlated with ~HDDs/~CDDs, subsetting / faceting by these
****''minor'': coordinated / aligned weather charts across time facets
****''minor'': differential charts: comparing absolute and relative differentials of this year:last year or actual:baseline; use of [[horizon charts|[[horizon charts|http://timelyportfolio.blogspot.ca/2012/08/horizon-on-ggplot2.html]] or heat maps
****''minor'': interactive toolips / higlighting and selection, details-on-demand, alternatives to raw ggplot: assessing viability of [[rCharts|http://ramnathv.github.io/rCharts/]] (with [[rickshaw.js|http://code.shutterstock.com/rickshaw/]] and [[leaflet|http://leafletjs.com/]] integration) , [[rVega|https://github.com/metagraf/rVega]] (R wrapper for [[Trifacta|http://www.trifacta.com/]]'s [[vega|https://github.com/trifacta/vega]]), [[rHighcharts|https://github.com/metagraf/rHighcharts]] (R wrapper for [[highcharts.js|http://www.highcharts.com/]], toolkit currently used in Pulse Energy Manager for interactive time-series visualization), [[RStudio|https://github.com/rstudio/rstudio]]'s [[ggvis|https://github.com/rstudio/ggvis]] (ggplot + vega + shiny)
**demo Friday
*CHI DC: notification delayed to as late as Dec 20
!!11 December
*email JM / CT re: TOCHI paper next steps
*(Pulse until 3:30pm):
**emailing DH re:
**portfolio sandbox: code refactoring, tweaking of bump and heatmap charts; data functions factored out into separate file; horizon chart experimentation; converted server.R query() functions to data table syntax
*meeting w/ TM @ Wicked Cafۛminutes|TM-13.12.11]]
!!12 December
*Pulse:
**portfolio sandbox: aesthetic improvments to visualization sketches for Friday demo, initial work on loading tags for a portfolio (JC / ~McGill sent initial tag set for portfolio), normalization and aggregation for box plots; added function for dynamic plot height for bump and heatmap plots
**meeting w/ Pulse's SJ to discuss portfolio visualization
***weather normalization and HDD/CDD comparison; better to compare baseload or midnight consumption; best at week or month level, too much noise at day / hour level, difficult to align weekends, holidays
***"time-over-time comparisons": superimposed load profiles or curve bundling to show typical loads (alternative: <<cite vanWijk1999 bibliography:Bibliography>> paper: [[Cluster and Calendar based Visualization of Time Series Data|http://www.win.tue.nl/~vanwijk/clv.pdf]].. van Wijk and E. R. van Selow, Proc. InfoVis 鬠p 4-9.); peak and baseload separation
*~EuroVis review requests x3
!!13 December
*Pulse:
**Friday demo: portfolio sandbox
**portfolio sandbox: filtering by tags (JC / ~McGill sent revised tag set for portfolio with zone and distribution tags); tag upload and formatting (long/wide formats, list / table format); general code cleanup and consistent formatting for legibility; rethinking data table organization for metadata, tag, and performance data;
!!To do:
*Overview
**new [[story w/ Overview re: food stamp website|http://www.wral.com/records-dhhs-downplayed-food-stamp-glitches/13173174/]]
**to do: email JS re: student study follow-up, email Jigsaw team re: scalability
*[[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!16 December
*Overview
**emailing TD, Overview user who published a story last week as a result of using Overview
*GRAC meeting re: NSERC and affiliated fellowships, fast-tracking review process for ~25 applicants by end of January
*Pulse:
**portfolio visualization: restuctured data tables (meta.dt, performance.dt, spacetags.dt); melting resource variables to facilitate stacked bar charts; aggregation and filtering by tag; filtering before joins; more tables for debugging;
***to do: fix aggregation for boxplots
!!17 December
*Pulse:
**portfolio visualization: refinement of tagging and aggregation; hacking about with replicating [[LineUp|lineup.caleydo.org]] in ggplot, to little success (faceting multiple chart types);
**meeting w/ KT
***portfolio visualization; visual indicators of tags for non-aggregated spaces or for spaces aggregated by some other variable
***alternatives to HDD/CDD normalization: comparison of baseload, demand at midnight (or midnight - 4am)
!!18 December
*CT+JM TOCHI paper revisions
*at UBC in AM, ~EuroVis review printing and picking up January uPass
*meeting w/ TM @ Wicked Caf୩nutes
!!19 December
*Pulse:
**portfolio visualization: fixed filter bug when selecting multiple options from list; adjusted API call to hour granularity; rethinking stacked / dodged / faceted bars for multiple resources: alpha and faceting by years; fixed aggregation and normalization for box plots
!!20 December
*Pulse:
**portfolio visualization: added filtering and selecting with map visualization, ensured compatibility with a portfolio without tags, UI fixes.
*CHI DC rejection, #264 on CHI SV waitlist
!!To do:
*[[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!23 December
*Pulse:
**portfolio visualization: added comments, hourly portfolio data now possible (though terribly slow for 2 years worth of data); discovered that API returns some duplicate records which complicate data table joins; metric area conversion (useful for drawing area-scaled space glyphs on map), began work on differential heatmaps
!!24 December
*Pulse:
**portfolio visualization: implemented differential heatmaps
*(leaving early for Christmas eve errands)
!!25 December
*(Christmas day holiday)
!!26 December
*(Boxing day holiday)
!!27 December
*(working from home)
*GRAC primary evaluation
*~EuroVis reviews (reading)
*Pulse:
**portfolio visualization: some experimentation with [[rCharts|http://ramnathv.github.io/rCharts/]]: interactive bar charts (no faceting); received updated space-tag list from UCB's KN
*some R data munging and analysis of [[P4K|http://pitchfork.com/]] Top Tracks data: multidimensional data set with artists, tracks, years, ratings, and rankings
!!To do:
*~EuroVis reviews x2
*TOCHI CTOC draft editing
*[[vega|https://github.com/trifacta/vega]], [[rickshaw.js|http://code.shutterstock.com/rickshaw/]], processing tutorials
!!References:
<<bibliography>>
!!30 December
*Pulse:
**portfolio visualization:
***initial success with [[LineUp|http://sgratzl.github.io/paper-2013-lineup/]] plots in ggplot; solution: attempt alternative to faceting, geom_rect with absolute quantities normalized to 0-1 scale, plotted with xmin,xmax,ymin,ymax to existing ~LineUp plot (geom_line + geom_text); using ordered intervals for geom_line, converted numeric intervals for geom_rect (resulting in legible axes and ability to fitler sorted factor intervals (seasons, months, weekdays)); transparent static geom_rect annotations that preserve legibility of geom_text in plot margins; removed previous ~LineUp plot attempting using faceting
***~LineUp plot height function: static height multiplied by # of facets
***attempting to fix [[leaflet-shiny|https://github.com/jcheng5/leaflet-shiny#]] ([[demo|http://glimmer.rstudio.com/jcheng/leaflet-demo/]]) signif lat/long rounding error (apparently only 6 signif #s are used, including 'mbol), which leads to noticeable precision errors when plotting spaces with longitude > -100
!!31 December
*Pulse:
**portfolio visualization:
***continued improvement to ggplot implemnentation of ~LineUp plots; revised ranking scheme (ties results in max rank, not average or min); line width and bar+line alpha mapped to a space's rank range over all visible intervals, such that spaces that change rank are more salient than those that do not; bar width redundantly encoded with line width such that highly-ranked spaces are more salient than low-ranked spaces; experimentation with different alpha/size scales and limits, preventing complete transparency or inability to separate line with from bar width;
***used [[directLabels|http://directlabels.r-forge.r-project.org/]] to produce aligned, legible labels for first //AND last// intervals in ~LineUp plot
***continuing to delve into [[leaflet-shiny|https://github.com/jcheng5/leaflet-shiny#]] signif lat/long rounding error
***converted intervals to sorted factors for ~LineUp plot, including weeks (which were numeric and were thus confusing ggplots' x-axis rendering when filtering)
***experimented with text week labels (e.g. 2013-12-29 - 2014-01-04); works for ~LineUp plot, but breaks differential plot and heatmaps (which compare week n,year x to week n, year y)
***experimenting with [[rCharts|http://ramnathv.github.io/rCharts/]]' version of leaflet wrapping; lat/long precision error occurs here too: apparently the solution involves converting lat/long objects to geoJSON format (as opposed to plotting directly as addCircles), as in [[this example visualizing bike sharing networks|http://ramnathv.github.io/bikeshare/]]
***~LineUp plot height function revised to resize based on # of unique ranks
*(leaving early for NYE errands)
!!01 January
*(New Years' day holiday)
!!02 January
*Pulse:
**portfolio visualization: continued improvement of ggplot implementation of ~LineUp plots: attempted to use a stable categorical colour mapping to spaces using [[color brewer|http://colorbrewer2.org/]], but this requires foreknowledge of how many spaces are to be shown, and more than 12 spaces results in repeated colours, reducing utility of stable colour mapping
***began work on interactive bump plot using [[rCharts|http://ramnathv.github.io/rCharts/]] wrapper for [[polycharts.js|http://www.polychartjs.com/]] (javascipt library motivated by grammar of graphics); limitations: no geom_rect analog, line size and opacity scaling control not as flexible as ggplot's size_scale_manual and alpha_scale_manual controls; used points and point size as proxy for geom_rect; limitations: hover and tooltips only work for points, not lines; x axis must be quantitative, not ordered factor, thus limiting the legibility of intervals
****inspirations: [[rCharts implementation of interactive pcoords|http://ramnathv.github.io/rChartsParCoords/]], [[rCharts + polycharts.js implementation of NYT baseball strikeout graphic|http://ramnathv.github.io/rChartsNYT/]], [[polycharts.js wiki|https://github.com/Polychart/polychart2/wiki]], [[|Interactive Visualizations with rCharts and Shiny|http://ramnathv.github.io/rChartsShiny/]]
***~LineUp plot height function revised to resize based on maxuimum # of unique ranks across all intervals (best solution thus far, removing lots of whitespace)
***attempted [[ggvis + shiny|https://github.com/rstudio/ggvis/blob/master/demo/apps/basic/server.r]] integration (ggvis is [[RStudio/hadley|https://github.com/rstudio/rstudio]]'s wrapper for [[vega.js|http://trifacta.github.io/vega/]]), however existing reactive data functions resulted in error with no clearly observable cause, likely something to do with UI selection inputs; [[ggvis google group discussion|https://groups.google.com/forum/#!forum/ggvis]]
***continuing to delve into [[leaflet-shiny|https://github.com/jcheng5/leaflet-shiny#]] signif lat/long rounding error; attempting to convert lat/long objects to geoJSON resulted in error ([[leaflet-rcharts|http://ramnathv.github.io/bikeshare/]] integration is poorly documented), [[leaflet-shiny|https://github.com/jcheng5/leaflet-shiny#]] integration apparently doesn't [yet] support geoJSON objects; switched to [[stamen.com/toner-lite|http://maps.stamen.com/toner-lite/#12/37.7706/-122.3782]] map tiles for improve saliency of space circles
***fixed UBC portfolio nrow(unique(meta.dt$spaces))) w(unique(performance.dt$spaces))), several aggregate spaces had no lat/long (removed from portfolio)
**DH sent Economist's [[Global house prices: Location, location, location|http://www.economist.com/blogs/dailychart/2011/11/global-house-prices]]: interactive, animated, multi-attribute time-series with rescaling y-axis, categorical and temporal selections
*GRAC primary evaluation x2
!!03 January
*Pulse:
**investigating feasibility of [[vega.js|http://trifacta.github.io/vega/]]: result, vega is intended for generating declarative visualizations, but not declarative interactions: interactions currently limited to hover (ggvis has linking and brushing)
**checking out [[+datavisualization.ch|http://selection.datavisualization.ch/#]] and [[Andy Kirk|http://www.visualisingdata.com/]]'s [[list of visualization resources|http://www.visualisingdata.com/index.php/resources/]]
**alternatives: likely [[raw D3|http://d3js.org/]] or [[processing.js|http://processingjs.org/]]: highlighting and selection, details-on-demand time-series charts
**wiki journal posts
**portfolio visualization: progress on rChart implementation of bump plot / lineup plot;
!!To do:
*Pulse portfolio visualization:
**new design slide deck for KN/UCB, JC/~McGill
**implement ~LineUp plot in [[D3|http://d3js.org/]] or [[processing.js|http://processingjs.org/]]
**reconsider hourly data; find workaround for slow loading and/or reduce size of data set (e.g. 2012-01-01 - 2013-12-31)
**using existing leaflet map and [[rCharts|http://ramnathv.github.io/rCharts/]] implementation of bumpplot to explore space highlighting and selection, details-on-demand time-series charts and/or differential horizon charts
**implementing interactive heatmaps and diffmaps with [[rCharts|http://ramnathv.github.io/rCharts/]]
**calendar visualization 졠van Wijk + van Selow '99 / [[d3 calendar implementation|http://bl.ocks.org/mbostock/4063318]]
**baseline / HDD+CDD normalized comparisons
**indicators of missing data
**visibility of tags in tooltips / details-on-demand
*~EuroVis reviews x2
*TOCHI CTOC draft editing
*check out [[animint|http://sugiyama-www.cs.titech.ac.jp/~toby/animint/index.html]]: R package provides limited interactivity and animation for charts (by [[Toby Hocking|https://github.com/tdhock]], Tokyo Inst. Technology)
!!References:
<<bibliography>>
!!06 January
*Pulse:
**portfolio visualization: added timeseries small multiple line plots; resized sidebar, began work on outlier scaling in diffmaps (what's the right cap?x)
!!07 January
*Pulse:
**portfolio visualization: total and electricity-specific baseloads; 4h interval granularity; exact match filtering; filter by space name via list; day of month filtering; fetched and tidied 4h-interval portfolio data from API for mcgill, ucb, ubc, ssd, sandbox portfolios
!!08 January
*Pulse:
**portfolio visualization: bug fixes (e.g. tag filtering bug introduced metadata filering refacroing, which included a cartesian joins without resetting keys); code commenting; removed debug table output; faceting lineup plots by year; dodging bar charts (rather than stacking); refactoing ranking and metadata filtering; plot sizing;
**helped CL re: R visualization packages beyond ggplot2: [[ggivs|https://github.com/rstudio/ggvis]], [[rCharts|http://ramnathv.github.io/rCharts/]], [[leaflet-shiny|https://github.com/jcheng5/leaflet-shiny]], [[ShinyDash|https://github.com/trestletech/ShinyDash]], [[rVega|https://github.com/metagraf/rVega]], rHighCharts (now rolled into rCharts), [[GoogleVis|http://cran.r-project.org/web/packages/googleVis/index.html]], [[clickme|https://github.com/nachocab/clickme]]
*Inquiry into Mitacs pay schedule
!!09 January
*Pulse:
**portfolio visualization: comments in code and in UI describing various chart types; fixed name filtering and coordination with map selection (no more full text search, option list instead); consumption multiplier for boxplots
**company standup
**began work on new design slide deck for JC/~McGill, Pulse
!!10 January
*WFH (GRAC meeting cancelled)
*CTOC TOCHI manuscript edits
*meeting w/ TM @ Arbutus Coffee 4pm: minutes
!!To do:
*Pulse portfolio visualization:
**finish new design slide deck for KN/UCB, JC/~McGill, JC (Pulse)
**implement ~LineUp plot in [[D3|http://d3js.org/]] or [[processing.js|http://processingjs.org/]]
**using existing leaflet map and [[rCharts|http://ramnathv.github.io/rCharts/]] implementation of bumpplot to explore space highlighting and selection, details-on-demand time-series charts and/or differential horizon charts
**implementing interactive heatmaps and diffmaps with [[rCharts|http://ramnathv.github.io/rCharts/]]
**indicators of missing data
**visibility of tags in tooltips / details-on-demand
*GRAC primary reviews x 1, find secondary reviewers x 2
*~EuroVis reviews x2
*TOCHI CTOC draft editing
*check out [[animint|http://sugiyama-www.cs.titech.ac.jp/~toby/animint/index.html]]: R package provides limited interactivity and animation for charts (by [[Toby Hocking|https://github.com/tdhock]], Tokyo Inst. Technology)
!!References:
<<bibliography>>
!!13 January
*Pulse:
**portfolio visualization: wrote github-flavoured markdown for project (README.md); worked on slide decks for ~McGill and UCB;
**meeting w/ KT to discuss project
*Overview:
**JS to interview journalist user tomorrow
!!14 January
*Pulse:
**checking out [[g3plot|https://github.com/alexbbrown/g3plot-1]] (extensible Shiny + D3 application)
**portfolio visualization: completed and sent slide decks for ~McGill and UCB, Pulse devs; sent out slides to RM, CC, JC, requested meeting, README.md improvements
**analytics planning meeting
**Energy Manager planning meeting
*GRAC reviews x 2
*~EuroVis reviews x2
!!15 January
*Pulse:
**portfolio visualization: begin implementation of ~LineUp plot in [[D3|http://d3js.org/]] or [[processing.js|http://processingjs.org/]]
**fixed bug re: unnecessary tag list join;
**meeting w/ Pulse's JC to discuss portfolio visualization sketches
**deployment of portfolio visualization sketches on ~RStudio Server w/ UCB data
!!16 January
*Pulse:
**portfolio visualization: meeting w/ Pulse's RM +CC, meeting w/ FL to discuss portfolio visualization sketches; migration to ~RStudio server (input UI initialization); fixed bugs re: reenabling the tag filter (it had been disabled for testing), input$resource_select (only electricity was working)
**deployment of portfolio visualization sketches on ~RStudio Server w/ UCB data, in sync w/ git repository
**updating design slides with notes from meetings
!!17 January
*(On campus)
*Meeting w/ TL (prospective ~PhD student) @ UBC
*GRAC meeting 11am
*~EuroVis reviews
*(dentist appointment)
*(pick up car at mechanic)
*~InfoVis group dinner
!!18-19 January
*~EuroVis reviews x2
!!References
*relevant D3 [[mbostock blocks|http://bl.ocks.org/mbostock]]: [[gradient bump|http://bl.ocks.org/mbostock/6059532]], [[sortable table with bars|http://bl.ocks.org/mbostock/3719724]], [[sorted bar chart|http://bl.ocks.org/mbostock/1389927]], [[sortable bar chart|http://bl.ocks.org/mbostock/3885705]], [[diverging calendar heatmap|http://bl.ocks.org/mbostock/4063318]] ([[source|view-source:http://bl.ocks.org/mbostock/raw/4063318/]]), [[slope graph|http://bl.ocks.org/biovisualize/4348024]] (by [[biovisualize|http://bl.ocks.org/biovisualize/]])
!!20 January
*Pulse:
**portfolio visualization: generate next-iteration mockups for demoing to ~McGill's JC / continue implementation of ~LineUp plot in [[D3|http://d3js.org/]]; added max/min points in time series line plots
**literature search: time-series scagnostics
**weekly standup meeting
*~EuroVis reviews x2 final pass
!!21 January
*Pulse:
**portfolio visualization: corrected weekday alignment by year (sunday is first, saturday is last), date filter correction; added calendar-based heatmaps with day-of-month labels on individual cells; constant line width for ~LineUp plots
*~EuroVis reviews due 4pm
!!22 January
*Pulse:
**portfolio visualization: meeting w/ ~McGill's JC and Pulse's FL via Skype to discuss portfolio visualization sketches; fixed duplicate metadata join bug; project folder tidying and README update; re-enabled week filter;
!!23 January
*Pulse:
**portfolio visualization: adjusted map zoom / bounds to accommodate geographically dispersed portfolios (Entrk portfolio spans BC); wrangling the Entrk portfolio
**meeting w/ Pulse's JC to discuss portfolio visualization sketches
**meeting w/ KT to discuss portfolio visualization sketches: drill-down workflows and reconciling different workflows, calendar-based heatmaps, high-fidelity prototyping; BG users; 5 Whys?
!!24 January
*(at Pulse until 2pm)
*Friday demo: calendar-based heatmaps for portfolio analysis
*meeting w/ Pulse's HR to discuss portfolio visualizations; getting BG data into the prototyping environment
*Pulse's BC requested access to portfolio visualization sketches
*GRAC meeting canceled
*meeting w/ TM: [[minutes|TM-14.01.24]]
!!26 January
*Arranging meeting w/ BG analyst lead; preparing and sending portfolio visualization sketch slides
!!References:
*closed stackoverflow thread on [[the "most underused data visualization"|http://stackoverflow.com/questions/2076370/most-underused-data-visualization]] including calendar heatmaps et al.
*[[time series calendar heatmap in ggplot|http://www.r-bloggers.com/ggplot2-time-series-heatmaps/]] [[calendarHeat.R source|https://github.com/jbryer/makeR/blob/master/R/calendarHeat.R]]
*[[mbostock block|http://bl.ocks.org/mbostock]]: [[diverging calendar heatmap|http://bl.ocks.org/mbostock/4063318]]
<<bibliography>>
!!27 January
*Pulse:
**portfolio visualization: 2h conversation w/ UCB's KN: lots of feedback re: workflows and design ideas; notes from last week's conversation with ~McGill's JC; preparing for tomorrow's BG demo
**weekly standup meeting
*~EuroVis reviews disussion and revising review
!!28 January
*Pulse:
**portfolio visualization: demoing to BG's AC (analytics lead), with HR; sent updated screenshots; making notes; continuing notes / workflow documents from JC, KN interviews
**Analytics HQ brainstorming w/ BC, remote communities manager
**Analytics HQ visualization brainstorming re: visualization, w/ PT, CL, JSB, AD, CP
*responding to LT (collaboration request via SI re: food science bibliometric data)
!!29 January
*Pulse:
**portfolio visualization: consolidating / reconciling notes and workflows from JC, KN, AC interviews; setting up local laptop install for Pulse's JC with entrk portfolio (duplicate space detection?)
**Analytics HQ follow-up w/ BC, PT, CL
**[[BG Energy Information Dashboard|https://www.britishgasbusinessdashboard.co.uk/Dashboard/Index/1]]
!!30 January
*Pulse:
**portfolio visualization: consolidating / reconciling notes and workflows from JC, KN, AC interviews
**Analytics HQ / Energy Manager discussion (missed this)
**Analytics HQ / BG planning session
**meeting w/ KT:
***see load duration curves in Energy Manager; ~McGill's JC and UCB's KN know their buildings well, no surprise that rankings are secondary and mainly used for infrequent presentation; ranking may be more important for those who don't know their buildings as well.
*UBC CS Alumni Seminar @ UBC Robson Square: Rock Leung on the future of BI
!!31 January
*(at UBC)
*GRAC meeting
**GATS application reviews x2
*thesis proposal timeline: scheduling with TM, JM, RR
*meeting w/ TM: [[minutes|TM-14.01.31]]
!!03 February
*Pulse:
**portfolio visualization: consolidating / reconciling notes and workflows from JC, KN, AC interviews; emailing FL re: Sutter; scheduling meeting w/ BC, requested organization ID #s for loading into sketch environment; emailing HR re: 2nd BG contact; literature search re: Weaver, Stolte / Mackinlay / Hanrahan
**weekly standup meeting
*GRAC reviews and grad visit day coordination
!!04 February
*Pulse:
**portfolio visualization: feedback and workflow consolidation; retrieved Sutter portfolio JSON from API, not tst database
**2014 Dev team planning
*GRAC reviews and grad visit day coordination
!!05 February
*Pulse:
**portfolio visualization: feedback and workflow consolidation; meeting w/ BC re: tagging, portfolio exploration, tag consistency, sectors / verticals / associations (e.g. convenience store associations, 19K churches in Michigan), end-use disaggregation; BC to retrieve list of spaces / organizations as a portfolio to examine;
**company standup 9am
**Friday demo planning 4pm
*Meetup: [[Vancouver Data Visualization|http://meetu.ps/29dMFn]]: //Colourful Design: Colour for information visualization// (David Darvill, VIVA / SFU) and //Visual Analytics: How we got here// (~Jean-Sebastien Mercier, VIVA)
*GRAC reviews and grad visit day coordination
!!06 February
*Pulse:
**portfolio visualization: feedback and workflow consolidation slides complete, sent to TM; meeting w/ JC re: Entrk portfolio cancelled - postponed to give JC more opportunity to explore tool; Sutter portfolio metadata [[JSON parsing|http://www.inside-r.org/packages/cran/RJSONIO/docs/fromJSON]], conversion to compatible CSV, site tagging; US API calls not working (not sure why)
**analytics code sharing session
**from CL: Shiny [[react log debugging|http://www.inside-r.org/packages/cran/shiny/docs/showReactLog]]
*GRAC reviews and grad visit day coordination
!!07 February
*Pulse:
**portfolio visualization: prototyping, Sutter portfolio retrieval attempt #2 (success: editing the API call in Pulse's ~UtilsPKG to use org_id #s); limiting the number of items shown in ~LineUp plots; added histogram / density plots as alternative to box plots; inviting demo session to Sutter/Accenture team (scheduled for 11am Thu Feb 20); informal demo to Pulse analytics team's BG
**Friday demo
*GRAC meeting canceled (GRAC offers, visit day logistics)
*meeting w/ TM canceled: meeting next Wednesday instead (sent consolidation slides)
!!10 February
*(Family day holiday)
*Pulse:
**portfolio visualization: scheduled Sutter/Accenture demo Thu Feb 20 11am
*GRAC reviews x2 and grad visit day coordination
!!11 February
*Pulse:
**weekly dev standup
**portfolio visualization: prioritizing prototyping, attempting to track down PG&E religious institution portfolio data; meeting w/ KT
*GRAC organizing and grad visit day coordination
!!12 February
*Pulse:
**portfolio visualization: R sandbox prototyping (see list below)
*meeting w/ TM @ Blenz Broadway/Granville 1:30 - 3pm: [[minutes|TM-14.02.12]]
*GRAC reviews and grad visit day coordination
!!13 February
*Pulse:
**portfolio visualization: R sandbox prototyping (see list below)
!!14 February
*Pulse:
**portfolio visualization: R sandbox prototyping (see list below); brief meeting w/ BC to discuss portfolio analysis workflow
*GRAC meeting [via Skype]
*Overview: awaiting ~EuroVis paper acceptance decision (Feb 15, rejection!)
!!R sandbox prototyping - done this week:
*major code refactoring; limited # rows shown in all plots; provided option to show n best / n worst;
*heatmaps: added absolute differentials; added option to set ᰰ% differentials to NA (corrects a skewed gradient scale); added text labels
*box plots: added options for range capping and outlier filtering; added dodging for differential box plots and superimposed density plots; combined bar / box / density plots with histograms
*lineup plots: modified visual encoding to reduce line salience, increased bar salience (to reduce false positives); adjusted line positions; removed vertical faceting; added text labels
*portfolio map: added performance data to the map
!!References:
*from CL: [[Sending data from client to server and back using shiny|http://ryouready.wordpress.com/2013/11/20/sending-data-from-client-to-server-and-back-using-shiny/]], [[reference index for shiny|http://www.inside-r.org/packages/cran/shiny/docs]]
!!17 February
*Pulse:
**portfolio visualization: preparing Sutter Health portfolio demo for Accenture (slides / annotated screenshots); prototyping to-do list (below); scheduling meetings w/ BC, CC, JC
*Overview:
**~EuroVis review commenting / sent to co-authors
!!18 February
*Pulse:
**portfolio visualization: prototyping to-do list (below); showing data on the map (solution [[here|https://groups.google.com/forum/#!topic/shiny-discuss/1pguNdowN6E]]); comments from KT re consolidated user feedback:
>''KT'': //A few random, quick comments for you.//
>
>''KT'': //Very detailed and useful doc. I think this is a good starting point for [CC] and [RM]. Despite all the detail though, some of the visualizations will probably require looking at the code because I suspect that what the visualization is really showing isn't clear. A good example is the calendar heatmap: it's kind of described but not crystal clear what is being shown. Or at least it wasn't to me ... :-)//
Good to hear. I堳cheduled a time with Cailie next week, and included the slides in the meeting request. I젡lso be sharing with Reetu, though I imagine she wont around to seeing these until after her vacation.
I堢een working improving the interpretability of the heatmaps; a few small tweaks to the legend (adding units, discretizing the scale) and adding text labels to individual cells is a step in this direction.
It seems like months are the smallest useful granularity for heatmaps spanning a large time window; for narrower time windows, calendars would be likely be more appropriate.
I堡lso been working on streamlining the interface of the sandbox environment such that heatmaps and charts containing summary statistics (bar charts, box plots, or histograms), appear on the same page; their juxtaposition allows for quick cross-references between the more granular heatmaps and the less granular summary plots.
>''KT'': //The calendar heatmaps could be used in other ways too. E.g. show the magnitude of difference for each compared to the similar day in the previous year. This would allow quickly picking out "bad days". //
On my list of ideas to work on is extending the differential heatmap encoding to the calendars, which requires some date manipulation, such that weekdays line up with weekdays and weekends with weekends.
>''KT'': //Also, there could be a next level of detail where a user could click on a day and have a simple 24 hour visualization showing the heatmap by hour (or difference to the similar day in the previous year). Or even click on a month and show a row or column for each day with the hour-by-hour heatmap.//
Based on the response I堧ot so far, it seems like heatmaps are preferred for coarser granularities; their may be more effective visual encoding choices for smaller time slices (days, weeks). To date I堤iscussed with you the idea of having time series load profiles or sparklines appear in tooltips, but their are other choices that better reflect the cyclical properties of hours in a day or days in a week, such as [[polar area charts|http://upload.wikimedia.org/wikipedia/commons/1/17/Nightingale-mortality.jpg]].
>''KT'': //As we discussed, some ideas around how to reduce the number of filters and/or provide defaults would be useful. if we could provide a default "best practice" workflow but some way to click and get more filtering options I think we'd have a winner because both beginners and advanced users could interact with the data.//
It could be that we provide simple and advanced metadata filtering, I see tags, geographical location, and primary use as being 魰le�ile building type, area, occupant count, operating hours, and building age could be advanced filters. The map is also a filter: currently it only supports single space selection, but Ike to extend it with a drag-select / lasso-select operation that filters to the spaces in the selected region.
As for time filters, I৯ing to make weekend/weekday/both and both/daytime/nighttime/both be a set of radio buttons. I feel that there likely isnything wrong with the date filtering options in Energy Manager already: a custom date picker, the last 12 months, last season / quarter, the last 7 days.
>''KT'': //Also, some smarts around aggregation would be good so we aggregate at the right granularity for the time range of data.//
I agree. The time granularity selection should be automatic based on the range of the time window, not configurable as it is in my sandbox (though this has been helpful for prototyping). I젢e working on implementing this.
>''KT'': //Limiting what is shown is fine except I think we need to give users some way to page through the data or enter a site name and see where it ranks in the overall list. I suspect a usability study could be done on this topic alone!//
I젳how you this on Monday: I堣onfigured the sketch sandbox such that by default the portfolio is filtered to the top 10 spaces (or groups of spaces), based on the average value of the selected resource and quantity for the selected time window, in descending order. I堡dded options to change this size, as well as the sort order.
>''KT'': //The reference material for each of the workflows is great to have. Thanks for putting this together!//
Great. I hope it൳eful. I will continue to use these workflow descriptions as I continue to prototype (and eventually, for writing about these workflows in the context of a research paper).
>''KT'': //I hope this helps, and makes sense.//
Yes. Thanks. Have a good trip back to Canada.
!!19 February
*Pulse:
**portfolio visualization: prototyping to-do list (below): stacked area charts in conjunction with line plots; preparing for Sutter / Accenture demo; PG&E religious institution metadata from AH (16K buildings) - need to determine site IDs and verify if metadata is sufficient / whether interval data can be accessed via the Pulse API (database connection seems to be down)
*HCI GRAC meeting via Skype
!!20 February
*Pulse:
**portfolio visualization: Sutter Health portfolio demo for Accenture 11am (w/ FL, BA); compiling notes, adding to slides
***reading about inter-portfolio analysis: [[Energy benchmarking for commercial office building|https://www.nrcan.gc.ca/energy/efficiency/buildings/energy-benchmarking/3747]] (source: nrcan.gc.ca); Energy Star's [[Portfolio Manager|http://www.energystar.gov/buildings/facility-owners-and-managers/existing-buildings/use-portfolio-manager]] tool for benchmarking and reporting commercial buildings; [[CBECS: Commercial Buildings Energy Consumption Survey (2003)|http://www.eia.gov/consumption/commercial/index.cfm]] - US Energy Information Administration (US EIA)
!!21 February
*Pulse:
**Friday demos / Sochi Olympic Men's Hockey Semifinal
**portfolio visualization: prototyping to-do list (below)
**responding to KT email (see above)
*GRAC meeting cancelled
!!Done this week:
*R sandbox prototyping
**--lineup plots: legends and interval labels along top-- (restricting # of items facilitates this)
**portfolio map: added performance data to the map with same colour scale as heatmaps
**line plots: aggregate stacked area plot in conjunction with small multiples
**refactoring
!!References:
*[[Leaflet and Shiny - Color Control of Shapes|https://groups.google.com/forum/#!topic/shiny-discuss/1pguNdowN6E]] - Google Groups thread
*[[Leaflet.js reference|http://leafletjs.com/reference.html]]
*Google Groups thread on [[Leaflet in R Shiny|https://groups.google.com/forum/#!msg/shiny-discuss/V7WUQA7aAiI/gnlLIG8N2-QJ]], beginning with [[this post|https://groups.google.com/forum/#!msg/shiny-discuss/V7WUQA7aAiI/gnlLIG8N2-QJ]] regarding lat/long precision in leaflet-shiny
*[[rCharts + Rickshaw.js for interactive stacked area time series charts|http://timelyportfolio.github.io/rCharts_rickshaw_gettingstarted/]]
<<bibliography>>
!!24 February
*Pulse:
**portfolio visualization: need to determine site ~IDs and verify if metadata is sufficient / whether interval data can be accessed via the Pulse API (database connection seems to be down); UI consolidation or mockups;
**meeting w/ KT: minutes
**weekly standup meeting
!!25 February
*Pulse:
**portfolio visualization: meeting w/ BC re: workflows + Sutter portfolio; --meeting w/ CC + knowledge transfer--;
!!26 February
*Pulse:
*company standup 9am
**portfolio visualization: (see list below)
*meeting w/ TM: 4pm @ Wicked Cafۛminutes|TM-14.02.26]]
!!27 February
*Pulse:
**portfolio visualization: (see list below); --meeting w/ JC re: entrk portfolio--
**analytics code share
!!28 February
*Pulse:
**Big Idea Day
**portfolio visualization: (see list below); knowledge transfer meeting w/ JM + JN; knowledge transfer meeting w/ KT
*GRAC meeting (via Skype)
!!R sandbox prototyping done this week:
*--cross-filter coordination-- (not easily accomplished, replaced with "no result" message upon query)
*added normalization by degree days in addition to baseload comparisons
*date range filtering rather than strict within-year filtering; time window filtering consolidation; radio buttons for time-of-day analysis, weekend / weekday analysis
*heatmaps: extended differentials to calendar heatmaps; added options to control color gradient scale by removing outages, relative and absolute amounts
*line plots: sort by performance
*Pulse portfolio visualization to do:
**examine load duration curves in EM
**R sandbox prototyping
***--portfolio map--: [[choropleth maps for larger portfolios?|http://leafletjs.com/examples/choropleth.html]] (larger portfolios)
**JS high-fidelity prototyping:
***backtracking and breadcrumbs? cross-filter coordination; temperature data in tooltips
***heatmaps: sparkline / line plot tooltips, drill-down to small multiples; aligned and coordinated box plots or histograms with linked highlighting and selection;
***lineup plots: mouseover tooltips with sparklines or line plots, drill-down; value tooltips for selected series + interval
***portfolio map: using existing leaflet map and [[rCharts|http://ramnathv.github.io/rCharts/]] implementation of bumpplot to explore space highlighting and selection, details-on-demand time-series charts and/or differential horizon charts
***line plots: linked highlighting and selection, crosshair navigation; area-under-curve selection and cumulative quantity estimation
**indicators of missing data (missing data vs. missing usage?)
**visibility of tags in tooltips / details-on-demand
!!3 March
*Overview VIS revisions; pcs submission creation
!!4 March
*Overview VIS revisions
!!5 March
*Overview VIS revisions: token to TM
*MUX forum: OS research update on Haptic Sketching
!!6 March
*Pulse:
**portfolio visualization: meeting w/ KT, meeting w/ JC; preparing and sharing consolidated documentation, --preparing walkthrough video-- (deferred); Mitacs accelerate exit report
**resolving connectivity and access issues
!!7 March
*GRAC grad visit day
*CTOC interruption journal paper figure mixup
*CSGSA coffee house
!!8 March
*CTOC interruption journal paper figure mixup
*Pulse: Mitacs accelerate exit report and survey; sent to TM and KT for signatures
!!10 March
*(at home, ill)
*preparing proposal document, scheduling and organizing defense
*literature review and reference management, group website updates
!!11 March
*(at home, ill)
*downloading / reading about [[python/bokeh|http://bokeh.pydata.org/]]
*Pulse: submitting Mitacs final report
*Overview: creating HCD loops diagram adapted from Lloyd/Dykes 2011
*preparing proposal document
!!12 March
*Overview VIS revisions: token from TM to MB, MB to JS
*HCI@UBC: Profs. Moura Quayle and Antony Hodgson to talk about //Advances in Design Pedagogy//
*meeting w/ TM at UBC 2pm
*installing MS Office
!!13 March
*attempting to install Windows 8 in bootcamp partition (failed, no bootable disk); opted to use Parallels instead
*Overview VIS revisions: token from MB to JS
*Overview: retrieved 5 adoption papers from Lam 2012 survey
*proposal: scanning HL, DA's proposals; reading HL's proposal
!!14 March
*Pulse
**meeting w/ KT, OR, CC re: portfolio vis integration, documentation + KT mockups
**preparing walkthrough screen capture video
**VPN access, tusrtudio access / ~RStudio IDE access
**data.table package bug: 1.9.1 collapses intervals in Portfolio Viz
*GRAC meeting 11am (via Skype)
*//Visualizing Data// by Ben Fry (Processing book) arrived
!!17 March
*Pulse:
**configuring shinyapps.io (won't deploy from spark.rstudio.com / package issues)
**installing Parallels Desktop, Tableau desktop
*CTOC journal paper edits
*checking in w/ JS re: Overview manuscript
!!18 March
*installing Parallels agent for iPad
*updated ~OmniGraffle, licensed Parallels; submitted invoices
*Pulse: configuring shinyapps.io (won't deploy from localhost / cryptic error)
*Overview:
**[[adoption papers lit review|Information Visualization Evaluation: Meta-Analysis]] from <<cite Lam2011 bibliography:Bibliography>>: <<cite Gotz2009>>, <<cite Wang2007>>, <<cite Kincaid2005>>, <<cite McKeon2009>>,<<cite Robinson2008>>
*Tuesday Tea
!!19 March
*deploying Shiny apps to shinyapps.io (UI loads, but content doesn't, hmmm)
*reading [[No One Knows What the F*** They堄oing (or 襠3 Types of Knowledge�tp://jangosteve.com/post/380926251/no-one-knows-what-theyre-doing]] by [[@jangosteve|http://twitter.com/jangosteve]]
*reading <<cite Kandel2011a>> for Overview re: data wrangling: [[notes|Information Visualization Evaluation: Meta-Analysis]]
*MUX 2pm: PB research update
*meeting w/ TM (deferred until JS responds)
*reading <<cite Fry2008>>'s //Visualizing Data//, ch 1-2 / learning Processing
!!20 March
*Pulse
**configuring shinyapps.io (won't install from spark.rstudio.com / package issues)
**preparing walkthrough screen capture video
**preparing linking/brushing mockups; drill-down mockups
**reviewing Tableau, graphical histories
*Overview: email back/forth w/ JS
!!21 March
*meeting w/ TM + JS via Skype
*Overview: IEEEVIS abstract submission deadline
*Overview draft revisions
!!22 March
*Overview draft revisions
!!23 March
*Overview draft revisions
!References
<<bibliography>>
!!24 March
*Overview:
**manuscript revisions, token from JS
**sent draft to SI, HL
**read <<cite Chaney2012 bibliography:Bibliography>>: [[notes|References: Overview]]
*travel booking
!!25 March
*Overview:
**read <<cite Cao2010>>: [[notes|References: Overview]], literature review
**feedback from HL, SI
**meeting w/ TM @ Arbutus Coffee 1:30pm
**emailing case study journalist AP re: log files
!!26 March
*Overview
**manuscript edits
!!27 March
*Overview
**manuscript edits, token to TM
**AP log file, interview / screen capture, notes case study data analysis
!!28 March
*Overview
**reading <<cite Hevner2004>>, <<cite Gregor2013>> on design science: [[notes|References: Overview]]
*volunteer lunch / raffle
!!29 March
*Overview
**manuscript edits
!!30 March
*Overview
**manuscript edits
!References
<<bibliography>>
!!31 March
*Overview: revisions, read-aloud, submission
*proposal / VIS DC
!!01 April
*proposal / VIS DC
*Tuesday Tea
!!02 April
*proposal / VIS DC
*MUX visiting ~PhD student from ~UVic
!!03 April
*@Pulse
**recording / sharing video walkthroughs
**reference / video sharing
**meeting w/ KT, RM, CC, FL
***[[github.kamisama/cal-heatmap|http://kamisama.github.io/cal-heatmap/]]
***potential to get user feedback from BG EM users in coming months
***how to show individual within context of portfolio/group after landing at individual building from Pulse Check? immediate drill-down based on site selection; different landing pages depending on referring page?
***see [[Songza|http://songza.com/]]'s playlist control re: landing pages (lobby directory vs. receptionist metaphor)
***boxplots with no vertical in box (Tufte-esque boxplots as used in <<cite Goodwin2013a bibliography:Bibliography>>
***maps: do EM users really think spatially? what about pseudo-spatially as in ~BallotMaps (<<cite Wood2011>>)
**meeting w/ KT
***[[McGill Energy Project's Energy Map|http://mcgillenergyproject.com/map.php]] ([[source|https://github.com/mcgillenergy/energymap]])
***POV: plan / optimize (diagnose) / verify;
***monthly data vs. interval data
***hierarchical tagging interface based on gmail interface for tag groups
***time of day: operating hours may differ within portfolio; time zones may differ as well.
***weather normalization: simple on/off interface, no HDD/CDD config
**Tableau experimentation / feature exploration
**reading <<cite vandenElzen2013>>
***checking out [[qt/C++|http://qt.digia.com/Product/]] (implementation language used in <<cite vandenElzen2013>>)
***[[rCharts + D3|http://zevross.com/blog/2014/04/03/interactive-visualization-from-r-to-d3-using-rcharts/]]
!!04 April
*proposal / VIS DC
*TM practice talk 3:45
!!06 April
*proposal / VIS DC, sent to TM for feedback
!References
<<bibliography>>
!!07 April
*proposal
*reading JD's IV journal draft
*~InfoVis review requests x2
!!08 April
*Overview: PI response re: 7 scenarios
*proposal
**feedback from TM, response
**scheduling w/ RR, JM
*CTOC TOCHI draft iteration
*~InfoVis reviews x2
*Tuesday Tea
!!09 April
*proposal
*HCI@UBC
!!10 April
*@Pulse: proposal (portfolio viz aspects)
!!11 April
*proposal draft v2 complete, sent to TM for feedback
*follow-up re: twitter intern position
!References
<<bibliography>>
!!14 April
*proposal - integrating TM's feedback
*reading <<cite Albers2014 bibliography:Bibliography>>, collecting refs
!!15 April
*(departing for Florida)
*checking out Processing.py
*~InfoVis reviews x2
*proposal to JM, RR, TM
!!16 April
*(in Florida)
*Overview's NEWYORK case study story a finalist for Pulitzer
*MUX
!!17 April
*(in Florida)
!!18 April
*(in Florida)
*(Good Friday holiday)
!References
<<bibliography>>
!!21 April
*(in Florida)
*(Easter Monday holiday)
!!22 April
*(in Florida)
!!23 April
*(returning from Florida)
*MUX
!!24 April
*@Pulse
**meeting w/ JN re: Energy Manager home screen
**Sending dissertation proposal to KT
**checking out [[Lyra|http://idl.cs.washington.edu/projects/lyra/]] [[VDE|http://idl.cs.washington.edu/projects/lyra/app/]] ([[tutorial|http://vallandingham.me/make_a_barchart_with_lyra.html]])
**checking out Design Density's [[RAW|http://app.raw.densitydesign.org]] app
**learning [[basics of Axure|http://www.axure.com/learn]]
*Meetup: [[Deep Learning for High Performance Time-series Databases: Ted Dunning|http://meetu.ps/2dfHfC]] @ Hootsuite
!!25 April
*(picking up bicycle in Delta)
*(picking up Sun Run package at UBC)
*writing statement for //Evaluation of Visual Analytics// Workshop
!!28 April
*finishing statement for //Evaluation of Visual Analytics// Workshop
*Slides for proposal defense
*~InfoVis review 1 / 2
!!29 April
*VIS DC revisions based on proposal
*reference management
*~InfoVis review 2 / 2
*(phone setup)
*[[Meetup: Exploring the research toolbox: what to use, when and why|http://meetu.ps/2jCj5d]]
!!30 April
*~InfoVis review 2 / 2
*(phone setup)
*Pulse webinar
*send reminder to committee re: proposal defense
*submitted statement for //Evaluation of Visual Analytics// Workshop
*to do: read highlighted sections of TM book draft
!!01 May
*@Pulse
**monthly company standup
**~ShinyPortfolio down due to memory consumption; configuring SSH / VPN (uncertain if this works)
**rstudio.pulseenergy.biz updated to R 3.1, data.table 1.9.2, which affects time segmentation in ~ShinyPortfolio; manually installing Shiny 0.8 or data.table 1.8.8 results in error
**meeting w/ BH re: Pulse webinar
**meeting w/ KT re: next steps / interviews, logistics, discuss thesis proposal and debrief from last week's meeting w/ JN
**meeting w/ CC re: visualization brainstorming
!!02 May
*reading highlighted sections of TM book draft
*meeting w/ TM 2pm @ The Brown Dog, West 10th
!!05 May
*reference management
*prep for proposal defense
!!06 May
*prep for proposal defense
!!07 May
*thesis proposal 10am-noon
*VA eval workshop notification; travel booking
*~InfoVis reviews
!!08 May
*~InfoVis reviews due
*VIS DC poster draft, VIS DC submission
*LUNCH 11:30
!!09 May
*VIS DC poster draft, final touches on VIS DC submission
!!12 May
*~InfoVis review discussion
*[[Visualizing U.S. Daily Temperature Anomalies 1964-2013|http://visualizing.org/full-screen/331512]] by dandelany
*Pulse / portfolio viz design / prototyping
**experimenting with [[cal-heatmap.js|http://kamisama.github.io/cal-heatmap/]]
!!13 May
*musing on calendar heatmaps
*~InfoVis review discussion
*Talk on Action Research (Cooperative Method Development), Yvonne Dittrich, Tue, May 13, 1:30 pm, Kaiser 2020
*reading <<cite Hutchins1995 bibliography:Bibliography>>'s //Cognition in the Wild//
*slides for Eval / VA workshop
!!14 May
*GPS/SCARL Workshop V: Modelling Proportion and Count Data 10-noon
*DRITW reference gathering / DR data viz tasks w/ typology lens
*reading <<cite Hutchins1995 bibliography:Bibliography>>'s //Cognition in the Wild//
!!15 May
*meeting w/ TM 2pm on campus: [[agenda|TM-14.05.14]]
!!16 May
*GRAC wrap-up lunch
!References
<<bibliography>>
!!19 May
*(Victoria Day Holiday)
*(Travel to San Diego)
!!20 May
*VACCINE [[Science of Evaluation Workshop|14.05.20-21 - Science of Evaluation Workshop]] at UCSD, La Jolla California (8am - 5pm)
!!21 May
*VACCINE [[Science of Evaluation Workshop|14.05.20-21 - Science of Evaluation Workshop]] at UCSD, La Jolla California (8am - 1pm)
!!22 May
*(in San Diego)
!!23 May
*(in San Diego)
*(Travel to Vancouver)
!!26 May
*Pulse
**prototyping in [[D3|http://chimera.labs.oreilly.com/books/1230000000345]]: [[cal-heatmap|http://kamisama.github.io/cal-heatmap/]], [[mbostock|http://bl.ocks.org/mbostock]]'s [[Focus+Context|http://bl.ocks.org/mbostock/1667367#index.html]], [[mbostock|http://bl.ocks.org/mbostock]]'s [[Calendar View|http://bl.ocks.org/mbostock/4063318]], [[Bostock's VISBI '12 Workshop Slides|http://bost.ocks.org/mike/d3/workshop/]], [[Arrays in D3|https://github.com/mbostock/d3/wiki/Arrays]], [[nested keys|http://bl.ocks.org/phoebebright/raw/3176159/]], [[mbostock|http://bl.ocks.org/mbostock]]'s [[Small Multiples|http://bl.ocks.org/mbostock/1157787]], [[tjdecke|http://bl.ocks.org/tjdecke]]'s [[Day/Hour Heatmap|http://bl.ocks.org/tjdecke/5558084]] (inspured by [[Trulia|http://www.trulia.com/vis/tru247/]]'s time-of-day house-hunting visualization)
**meeting with JN re: Energy Manager mockups, tablet-friendly design; boxplots and color stock charts, heatmaps with juxtaposed and coordinated sparkline visualizations
!!27 May
*photocopying and submitting expenses for VACCINE Science of Evaluation VA workshop
*email to MT, SC, BL re: BELIV paper on empirical research at early stages of visualization design
*Pulse: responding to email thread about juxtaposing time-varying and statistical summary visualizations
*Overview: Reading <<cite Smith2014 bibliography:Bibliography>> on Hi顲chie: visualizing hierarchical topic models; contacting/responding to first author Smith
**reference management
**[[installing local version of Overview|https://github.com/overview/overview-server/wiki/Installing-and-Running-Overview]]
*(Driving AC to airport)
*completed my portion of annual progress report for FOGS, sent to TM
*changing CS passwords
!!28 May
*DRITW BELIV paper
**graffling
**literature search
*(at home, ill)
!!29 May
*10am ~Sung-Hee Park on //The Role of Information Visualization in Consumer Decision Making// in ICCS 238
*11am Mark Schmidt on //Tractable Big Data and Big Models in Machine Learning// in x836
*Lunch with SHP / meeting SHP
*Dinner with LUNCH group + SHP
*~Early-Stage Empirical Work BELIV paper
!!30 May
*feedback to JM on SHK
*Pulse
**[[officeofjane|http://bl.ocks.org/officeofjane/]]'s [[small multiple bar charts with tooltips|http://bl.ocks.org/officeofjane/7315455]]
*ethernet connectivity
*DRITW BELIV paper
!References
<<bibliography>>
!!02 June
*Pulse
**developed [[faceted color-stock heatmaps with aligned load profiles|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]]
**[[Jerome Cukier|http://www.jeromecukier.net/]] on [[Manipulating data like a boss with d3|http://www.jeromecukier.net/blog/2012/05/28/manipulating-data-like-a-boss-with-d3/]]
**meeting w/ JC + KT: relative/absolute scaling as global control; cost transformation in addition to consumption and area/weather normalization transformations; outlier flagging / peak demand flagging in color stock charts and in load profiles; date granularity selection and rounding up to natural date divisions (week, month, year boundaries);
**meeting w/ EMU team: site tagging, pagination vs. demand loading for large search results / large portfolios (what about alternative sort orders); tagging as basis for bookmarking or linking to search results; dev phase 2 (June) to include tile mode heatmaps, load profiles, temperature as secondary time series, pagination / bookmark shelf, home page to include site settings and alerts, replacing POV interface; BG as first adopter users; color stock charts max/min heights may be misleading - adjust such that max/mean/min tiles all have uniform heights?; outlier flagging; next steps: CUSUM charts and baselines
*Overview
**response from A. Smith re: [[Hi顲chie|http://decisive-ui.github.io/Hierarchie/#/about]]: [[A structured display of topics from discussion surrounding the MH-370 disappearance|http://decisive-ui.github.io/Hierarchie/#/]]
!!03 June
*Pulse: deploying [[faceted color-stock heatmaps with aligned load profiles|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] with sandbox portfolio data; email from SJ re: workflow + comparison model
*sending annual progress report to JP
*early empirical research BELIV paper outline slides - sent to MT, BL, SC
*Dept. Awards Tea
!!04 June
*reference gathering for Pulse
*emailing JS re: Hi顲chie
*Vis tasks for DR data BELIV paper outline slides
*MUX forum 2pm: CHI/GI/GRAND AGM conference roundup
!!05 June
*reading <<cite Suchman1995 bibliography:Bibliography>> on making work visible, on representations of work practices
*Vis tasks for DR data BELIV paper outline slides
*LUNCH meeting 11:30am
!!06 June
*Vis tasks for DR data BELIV paper outline slides
*~InfoVis first round reviews: paper conditionally accepted!
**adding comments to reviews
!References
<<bibliography>>
!!09 June
*Early Empirical Work BELIV paper meeting via Skype w/ SC, MT, BL
**outline and paper writing logistics / shared Dropbox folder with .docx in ACM format
*Pulse (but not at Pulse, because Pulsars at Loon Lake retreat)
**working on portfolio viz prototypes in d3: working on interaction and coordinated tooltips for small multiple load profiles: [[x-value mouseover|http://bl.ocks.org/mbostock/3902569]], [[zoomable area|http://bl.ocks.org/mbostock/4015254]]
*completing Science of Evaluation workshop reimbursement form
*Overview: camera ready / fast forward video / slides instructions (see email from J Heer)
**~InfoVis review comments
!!10 June
*Pulse
**working on portfolio viz prototypes in d3: working on interaction and coordinated tooltips for small multiple load profiles:
*Vis tasks for DR data BELIV paper outline slides
*Overview: PCS site is now accepting revisions
*MUX social at Miko Sushi downtown
!!11 June
*HS ~PhD proposal defense practice talk
*Science of Evaluation workshop reimbursement: Glacier file?
*reading [[Activity-centered design 㠳ome thoughts|http://ebiinterfaces.wordpress.com/2013/03/13/activity-centered-design-some-thoughts/]]
*meeting w/ TM at Grindstone Caf頴pm: [[agenda/minutes|TM-14.06.11]]
*sending Overview revision plan to JS
!!12 June
*Science of Evaluation workshop reimbursement: tax headaches, GLACIER, ITIN, IRS form W7
*LUNCH meeting 11:30am
*Vis tasks for DR data BELIV paper outline slides round 2
*[[EuroVis '14 design study session features no actual design studies|http://eurovis.swansea.ac.uk/program.htm]]
*leaving early to pick up Ana from airport
!!13 June
*Early empirical work BELIV paper
!References
<<bibliography>>
!!16 June
*empirical work BELIV paper: meeting w/ SC, BL, MT postponed
*Overview: revisions and cover letter (complete)
*(at home, dental problems, dentist 3pm)
!!17 June
*(at home, dental problems, dentist 1pm)
*Viz tasks for DR data pre-paper v2
!!18 June
*(at home, dental infection recovery)
*Viz tasks for DR data pre-paper v2
*MUX forum: MN (absent)
*meeting w/ TM: [[minutes|TM-14.06.18]]
*Overview revisions and cover letter sent to JS for feedback
*Viz tasks for DR data pre-paper v3 - sent to MS for feedback
*Purdue Eval / VA workshop tax clarification email
!!19 June
*(at home, dental surgery prep)
*empirical work BELIV paper: meeting w/ SC, BL, MT: BL to add 4th scenario; MB to work on prelim discussion / practical guidance; everyone to add to 'further reading' section; intro needs re-framing / example; next meeting 1pm Thursday June 26; external reader before submitting TBD?
*Viz tasks for DR data paper outline and figures added to ACM submission template
*(urgent dental surgery in PM)
!!20 June
*(Birthday, dental surgery recovery)
*Overview: [[new story|http://www.creditcards.com/credit-card-news/agreement-purchases-repossessed-1276.php]]
*Viz tasks for DR data paper writing; paper writing schedule with coordination with MS
!References
<<bibliography>>
!!23 June
*(at home, dental surgery recovery)
*Viz tasks for DR data paper writing
*email w/ SP, MA Planning Candidate, School of Community and Regional Planning re: viz resources
*[[DIS experience night|http://dis2014.iat.sfu.ca/index.php/program/#DISEXP]]: ~PechaKucha, Opening of Demos and ~P-WiPs at Renaissance, ~A-Level
!!24 June
*Viz tasks for DR data paper writing
*LUNCH meeting
!!25 June
*Viz tasks for DR data paper writing: token to MS/TM
*(bike lock replacement)
!!26 June
*BELIV deadline extended by 1 week
*empirical work BELIV paper
**paper writing
**meeting w/ SC, BL, MT postponed
*email from CM (VIVA CANVAS)
*Overview: watching [[#gbj14: What kind of software does journalism need?|http://vimeo.com/91270592]]
!!27 June
*empirical work BELIV paper (edits?)
*(post-op dentist appt 11:45)
*reading [[Visualizing Algorithms|http://bost.ocks.org/mike/algorithms/]] by Mike Bostock
*Pulse: upgrading R, packages; fixing data.table bugs introduced with update
*Overview: ~InfoVis revision deadline (awaiting JS updates?) 5pm
!References
<<bibliography>>
!!30 June
*Viz tasks for DR data paper writing
!!01 July
*(Canada Day - half day)
*Viz tasks for DR data paper writing: token to MS
!!02 July
*Updating [[ShinyFork|http://spark.rstudio.com/mattbrehmer/ShinyFork/]] and [[shinyapps.io instance|https://mattbrehmer.shinyapps.io/ShinyFork/]] (date parsing not working for the latter
*Pulse: Updating [[PortfolioViz Sandbox|http://spark.rstudio.com/mattbrehmer/ShinyPortfolio/]] and attempting to fix/redeploy [[shinyapps.io instance|https://mattbrehmer.shinyapps.io/PortfolioSandbox/]] (latter not loading correctly)
**emailing JN, KT re: meeting next week
*created Overview project page (not public yet)
*Viz tasks for DR data paper writing: token to SI, token to TM
*empirical work BELIV paper writing; meeting postponed
!!03 July
*empirical work BELIV paper writing, meeting w/ SC, BL 1pm; BL edits; creating paper on submission site; MB token
*LUNCH meeting (㊍) 1pm
!!04 July
*(GER vs. FRA)
*empirical work BELIV paper: token to SC
*Viz tasks for DR data paper writing: meeting w/ TM, edits and version sent to MB for external reading
*(CSGSA board game night)
!!07 July
*Viz tasks for DR data paper writing + submit
**reading <<cite SantosAmorim2012 bibliography:Bibliography>>
*empirical work BELIV paper writin + submit
*BELIV deadline 5pm
*Overview: email from JS re: future support for logging
!!08 July
*07.16 MUX talk slides
*TM ~BioVis keynote practice talk postponed until Thurdsay
*(BRA vs. GER 1pm)
!!09 July
*@Pulse, meeting w/ JN + JM re: portfolio viz review 10am
**[[EMU test environment|https://em.ca.pulseenergy.biz/]] - reviewing, making notes
**[[Stephen Few on Dual Axis Charts|http://www.perceptualedge.com/articles/visual_business_intelligence/dual-scaled_axes.pdf]], or <<cite Isenberg2011a>> on dual axis charts
**internal interviews to begin next week
*[[females in dataviz|http://stephanieevergreen.com/females-in-dataviz/]]
!!10 July
*LUNCH meeting (㊍) 12:30pm
*meeting w/ CG (VIVA / CANVAS) 2pm
*TM ~BioVis keynote practice talk 3:30
!!11 July
*Overview: ~InfoVis 2nd round reviews expected - accepted! [[website|http://www.cs.ubc.ca/labs/imager/tr/2014/Overview/]] updates, CV updates
*checking out [[CANVAS 2014|http://www.canvas2014.ca/index.php/schedule]] schedule
*Pulse: notes on EMU prototype
!References
<<bibliography>>
!!14 July
*[[InfoVis group website|http://www.cs.ubc.ca/group/infovis/]] updates
*IEEE VIS DC accepted; updates to DC paper due Jul 31
*notes on Pulse EMU design
!!15 July
*[[InfoVis group website|http://www.cs.ubc.ca/group/infovis/]] updates
*Pulse: EMU design feedback from SL 11:00, HR 3:30; notes
*[[twitter world cup visualization|https://interactive.twitter.com/wccompetitree/]]
!!16 July
*preparing slides for MUX talk
*MUX talk: MB on Pulse project
*Pulse: EMU design feedback from FL 4pm
!!17 July
*[[New Overview story in the Atlantic|http://www.theatlantic.com/technology/archive/2014/07/the-brilliance-of-louis-cks-emails-he-writes-like-a-politician/374034/]]
*--LUNCH meeting (㊍) 12:00pm--
*TM HOPE X practice talk 1pm
*Pulse: EMU design feedback from MS 3:30
!!18 July
*[[SIGGRAPH exhibition registration|http://s2014.siggraph.org/attendees/registration]] deadline
*edits to VIS DC proposal - video preview confirmation
*Overview
**edits to Overview camera ready (- student study in FW)
**[[Overview used to examine online predator's solicitation conversations|http://nextgenforensic.wordpress.com/2014/07/18/can-we-analyze-word-associations-in-online-solicitation-texts/]]
*Pulse: notes on EMU prototype
!!21 July
*Pulse:
**preparing EMU feedback notes
**EMU design feedback debrief 3:30pm
!!22 July
*Pulse: EMU design feedback debrief 2pm
*dual axis time-series with different units considered harmful?
!!23 July
*reading <<cite Aigner2011b bibliography:Bibliography>> on Bertin's indexing method
*Pulse: responding to KT's email re: EMU feedback
*meeting w/ TM 12:30: [[minutes|TM-14.07.23]]
*MUX: JD on CPSC 344/544
*checking out [[Text Visualization Browser: A Visual Survey of Text Visualization Techniques|http://textvis.lnu.se/]] by [[ISOVIS|http://cs.lnu.se/isovis/]]
*registered for SIGGRAPH electronic theatre
!!24 July
*[[VIS registration opens|http://ieeevis.org/year/2014/info/registration/conference-registration]]
*Overview video preview
*VIS DC submission edits + video preview
*LUNCH meeting 12:30
!!25 July
*reading <<cite Wickham2011>> on 40 years of boxplots
*Overview video preview
*VIS DC submission edits + video preview
!References
<<bibliography>>
!!28 July
*[[CANVAS2014|http://canvas2014.ca/index.php/program]]
*[[VIS DC Video Preview|http://people.cs.ubc.ca/~brehmer/proj/dc-brehmer.mp4]]
*CANVAS reception
!!29 July
*[[CANVAS2014|http://canvas2014.ca/index.php/program]]
*Pulse: preparing for TM meeting
!!30 July
*[[CANVAS2014|http://canvas2014.ca/index.php/program]]
*MUX: Huan Li ~MSc Essay presentation
*meeting w/ TM: [[minutes|TM 14.07.30]]
*meeting w/ TM + CC (UOIT)
!!31 July
*LUNCH meeting 11:30
*submitted reimbursement for ~DIS2014 Experience Night
*bullying and harassment course + certification
*checking out Autodesk's [[Paper Forager|http://www.autodeskresearch.com/paperforager/]]
*reading <<cite Kindlmann2014 bibliography:Bibliography>>
*(preparations for camping)
!!01 August
*(August long weekend Tofino vacation)
!References
<<bibliography>>
!!04 August
*(BC Day Holiday)
!!05 August
*DR Task Sequence BELIV paper conditionally accepted
*~Pre-Design Empiricism BELIV paper accepted
*checking out [[Caleydo Domino|http://caleydo.org/projects/domino/]]
*checking out [[Qualitative Research Methods|https://bitly.com/bundles/o_28ljdooe44/4]] - bundle by Jonathan Stray
*preparing for Pulse meetings
!!06 August
*EKOS survey on NSERC PGS
*UBC 2014W award acceptance
*(at Pulse in PM)
*meeting w/ JN re: boxplots
*--meeting w/ JC + JN on evaluating the EMU prototype-- postponed until more data available
*reading [[Distribution Displays, Conventional and Potential|http://www.perceptualedge.com/articles/visual_business_intelligence/distribution_displays.pdf]] by <<cite Few2014 bibliography:Bibliography>>
*[[simplified boxplots used in fivethirtyeight story|http://fivethirtyeight.com/features/the-odds-of-winning-the-indianapolis-colts-weather-challenge/]]
*[[MUX twiki software resources|https://bugs.cs.ubc.ca/cgi-bin/twiki/view/Imager/HCIResources?unlock=on]]
*bus to Seattle
*read <<cite Huron2014a>> ~InfoVis '14 pre-print on constructive representation
!!07 August
*in Seattle for [[See, Think, Design, Produce]] (Tufte, Corum, Munroe, Popova)
*lunch w/ JB (UBC Stats)
*bus to Vancouver
!!08 August
*STDP reimbursement package prep, [[STDP notes|See, Think, Design, Produce]]
*BELIV paper clarification w/ paper chairs, BELIV paper edits
*participated as pilot participant in AP/FT's study
!References
<<bibliography>>
!!11 August
*searching for flights, airbnbs for VIS in (November)
*cs.ubc grad/supervisor online survey
*BELIV paper revisions
**DR Vis Tasks: reading submitted version and reviews
**~Pre-Design Empiricism (PDE): token to MT
*checking out CANVAS slides
!!12 August
*BELIV paper revisions / DR Vis Tasks figure edits
*SIGGRAPH exhibition and electronic theatre
!!13 August
*BELIV meeting w/ --TM--, MS, SI via Skype 8:30
**MS recommends [[Understanding the New Statistics|http://www.amazon.com/Understanding-The-New-Statistics-Meta-Analysis/dp/041587968X]] by G. Cumming
*meeting w/ TM via Skype to recap earlier meeting re: BELIV
*--NJ ~MSc presentation 11:30am--
*MUX: PB ~MSc presentation 2pm
*BELIV paper revisions
*adding interesting papers from VIS 2014 to to-read
*checking out [[Remix of the Century|http://www.remixofthecentury.com/]]
!!14 August
*BELIV (DR Viz Tasks) paper revisions
*[[paper page for PDE BELIV|http://www.cs.ubc.ca/labs/imager/tr/2014/PDE/]] paper
!!15 August
*BELIV (DR Viz Tasks) paper revisions
*BELIV (PDE) paper revisions
!References
<<bibliography>>
!!18 August
*[[IEEE VIS papers + video previews posted|http://ieeevis.org/year/2014/info/overview-amp-topics/accepted-papers]]
*[[IEEE VIS tutorials posted|http://ieeevis.org/year/2014/info/overview-amp-topics/accepted-tutorials]]
*[[Task Typology revisited notes|Task Characterization: Meta Analysis]]
*reading and working through <<cite Fry2008 bibliography:Bibliography>> ch 3: [[Mapping|http://www.cs.ubc.ca/~brehmer/research/processing/ch3/]]
*BELIV DR Viz Tasks token to TM
!!19 August
*[[migrating processing work to processing.js|http://www.cs.ubc.ca/~brehmer/research/processing/ch3/]]
*switched to github student plan (5 private repos)
*Pulse: boxplot mouse-over sketching in d3, Tableau
!!20 August
*checking out [[keyvis.org|http://www.keyvis.org/]]
*d3.js refreshing: [[dashingd3.js tutorials|https://www.dashingd3js.com/table-of-contents]]
*MUX: KZ, AS
!!21 August
*BELIV DR Vis tasks revisions
*reading <<cite Mckenna2014>>'s design activity framework paper: [[supplemental materials containing table of 100 methods and definitions|http://mckennapsean.com/projects/design-activity-framework/suppl-mat.pdf]]
*Pulse: boxplot mouse-over sketching
*LUNCH
*meeting w/ TM re: BELIV / DR vis tasks: [[minutes|TM-14.08-21]], token to SI
*BELIV DR Vis tasks revisions: Table 1
!!22 August
*reading and working through <<cite Fry2008 bibliography:Bibliography>> ch 4: [[time series|http://www.cs.ubc.ca/~brehmer/research/processing/ch4/]]
*BELIV PDE edits: adding ~McKenna citation
*BELIV DR Vis tasks revisions: replacing many 'could's
*~VIS2014 registration, flight search
!References
<<bibliography>>
!!25 August
*BELIV DR Vis Tasks: removing 'pre-design vocabulary'
*Pulse: [[d3.js box plots|http://bl.ocks.org/mattbrehmer/12ea86353bc807df2187]]
!!26 August
*d3.js box plots continued
*downloading [[VIS pre-prints of interest|http://ieeexplore.ieee.org/search/searchresult.jsp?ranges%3D2014_2015_p_Publication_Year%26matchBoolean%3Dtrue%26searchField%3DSearch_All%26queryText%3D%28%28Visualization+and+Computer+Graphics%2C+IEEE+Transactions+on%29+AND+p_Issue%3A99%29&rowsPerPage=100&pageNumber=1&resultAction=ROWS_PER_PAGE#]]
!!27 August
*GST/HST tax issues
*chatting w/ OS re: Shiny, ~IPython Notebook, bokeh, ggvis, etc.
*helping AC review Vanier application
*d3.js box plots continued: [[box plot w/ brushing|http://bl.ocks.org/mattbrehmer/8be29724bdd7a63ff41d]]
**[[SO: linked highlighting selections|http://stackoverflow.com/questions/11206015/clicking-a-node-in-d3-from-a-button-outside-the-svg/11211391#11211391]]
*[[p5.js launched|http://hello.p5js.org/]]
!!28 August
*checking out [[#tgw2014|https://www.graphicalweb.org/2014/]]
*reading CTOC TOCHI reviews
*LC SPIN study on touch sensor for robot creature
*LUNCH
*meeting w/ JM, CT, (CJ?) re: CTOC TOCHI paper discussion
*d3.js box plots continued: [[color stock chart|http://bl.ocks.org/mattbrehmer/82cf72481d4753eca1cf]]
!!29 August - 31 August
*created a [[github repo for BrainFreeze|https://github.com/mattbrehmer/BrainFreeze]] (~MSc project software)
*Pulse: reading KT's persona summary
*learning [[p5.js|http://hello.p5js.org/]] and working on <<cite Fry2008 bibliography:Bibliography>>'s ch.5: connections and correlations: [[salary vs. performance|http://bl.ocks.org/mattbrehmer/4473009667985e1f893b]] ([[BF's Processing.js version|http://fathom.info/salaryper/]])
**learning about [[classes in javascript|http://www.phpied.com/3-ways-to-define-a-javascript-class/]]
*submitting BELIV revisions
!References
<<bibliography>>
!!01 September
*(Labour day holiday)
!!02 September
*paid UBC fall term fees
*BELIV: PDE paper uploaded on Saturday not listed on CMT, emailed HL and BELIV organizers
*MUX buddy signup for Oct 1
*responding to Palantir recruiter
*Pulse: responding to KT, JN re: EMU personas and design (response from JN re: wireframes)
*reading <<cite Sacha2014 bibliography:Bibliography>>: [[notes|Task Characterization: Meta Analysis]]
*Orientation: student course pitch + pizza, 4-6pm in x836
*reviewing / completing tutorial on using the [[p5.dom library|https://github.com/lmccart/p5.js/wiki/Beyond-the-canvas]]
!!03 September
*working on juxtaposed focus + context boxplots
*Orientation: BBQ 5:30 at Spanish Banks
!!04 September
*(Car insurance renewal)
*reviewing VCGS scholarship statement for AC
*LUNCH
*completed [[juxtaposed focus + context boxplots|http://bl.ocks.org/mattbrehmer/287e44c9a12151967874]] (currently working in Chrome, Safari, but not Firefox)
!!05 September
*~InfoVis group meeting 10:30
*ML seminar (A Fyshe: //The Semantics of Phrases and Sentences in the Human Brain//) 11 x836
*CPSC 547 guest lecture talk prep: [[Visualization Design Resources]]
*meeting w/ TM 2:30: [[minutes|TM-14.09.05]]
!!To do:
*Overview talk prep
*VIS DC talk prep
*publish ggplot implementation of ~LineUp as a gist / blog post / Shiny App (later)
!References
<<bibliography>>
!!08 September
*[[Visualization Design Resources]] moved to [[InfoVis group website|http://www.cs.ubc.ca/group/infovis/resources.shtml]]
*reviewing VCGS scholarship materials for AC
!!09 September
*Apple keynote: ~iPhone 6, Watch, etc.
*Pulse EMU: can't login to portfolio accounts
**reviewing KN's persona updates / feedback from client services team
**Pulse: reviewing EMU design w/ JC, CC, JN, 1:1 w/ JN to review boxplot designs / mockups
**updated [[juxtaposed focus + context boxplots|http://bl.ocks.org/mattbrehmer/287e44c9a12151967874]]: fixed firefox dimension bug
*VIS DC talk prep: reviewing thesis proposal slides
*reading <<cite Satyanarayan2014a bibliography:Bibliography>>, trying out Lyra
!!10 September
*Lyra tutorials, creating visualizations: gold medals, NYT driving
*watching [[Bret Victor|http://worrydream.com/DrawingDynamicVisualizationsTalkAddendum]]'s [[Drawing Dynamic Visualizations|http://vimeo.com/66085662]] - Feb 1, 2013
*HCI@UBC: LC (Nursing), GM (ECE), LN (~iSchool), AS (CS)
*renewing student card
*MUX: CPSC 544 student buddy / paper discussion initial meeting
*VIS DC talk prep
!!11 September
*[[InfoVis group resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]] updates, now w/ thumbnails
*LUNCH
*VIS DC talk prep
*BELIV final notification expected: [[PDE|http://www.cs.ubc.ca/labs/imager/tr/2014/PDE/]], DR Vis tasks accepted, updated websites, created [[paper page for dr vis tasks|http://www.cs.ubc.ca/labs/imager/tr/2014/DRVisTasks/]]
!!12 September
*VIS DC talk prep
*reviewing <<cite Satyanarayan2014a bibliography:Bibliography>>
*~InfoVis group meeting 1pm: discussing <<cite Satyanarayan2014a bibliography:Bibliography>> on Lyra
*meeting w/ TM: 4pm
!References
<<bibliography>>
!!15 September
*copyright transfer for BELIV papers, adding copyright info to Fig 2.
*request for feedback from ~InfoVis group on [[InfoVis Group Visualization Design Resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]], fixed cross-references, h2 link CSS
*emailing JS re Overview practice talk
*VIS budget to TM
*VIS ~FFs
*Pulse EMU: notes from JC feedback session
*DC talk prep
!!16 September
*DC talk prep
*~MSc thesis presentation: //Managing Updates and Transformations in Data Sharing Systems// by Arni Thrastarson, 10am in x836
*BELIV PDE edits
*CHI draft review for MUX
!!17 September
*CHI draft review for MUX
*DC talk prep
*MUX: CHI draft review 2pm
*MUX: meeting w/ 544 buddy
!!18 September
*--LUNCH--
*~InfoVis group retreat day 1: prior work, (lunch, dinner)
!!19 September
*~InfoVis group retreat day 2: MB DC practice talk, lunch, future work, dinner
!References
<<bibliography>>
!!22 September
*email from SS (UBC resource mgmt, environmental studies) re: R
*CPSC 547 guest lecture by B. Shneiderman: //Interactive Visual Discovery in Temporal Event Sequences: Electronic Health Records and Other Applications//
**Joshua W. Shenk's [[Powers of Two|http://www.amazon.ca/Powers-Two-Finding-Innovation-Creative/dp/0544031598]]
**Chris Alexander's [[Pattern Language|http://www.patternlanguage.com/]] ("Light from Two Sides")
**[[DataKind.org|http://www.datakind.org/]]: " tackling the world's biggest problems through data science."
*meeting w/ B. Shneiderman, EH, JF:
**[[Simon Buckingham Shum on argument visualization|http://kmi.open.ac.uk/publications/member/simon-buckingham-shum]]
**[[TimeSearcher: Shape Searcher Edition|http://link.springer.com/chapter/10.1007%2F978-1-4471-2804-5_17]]
**[[MIT Simile project|http://simile.mit.edu/wiki/SIMILE:About]]
**Continuum (Southhampton)
*prep for CPSC 547 guest lecture on vis design resources
*post-CHI deadline beers
*MUX CPSC 544 grad buddy paper guidance
!!23 September
*[[Further validation of the task typology through usage|http://dare.uva.nl/cgi/arno/show.cgi?fid=529854]].
*prep for CPSC 547 guest lecture on vis design resources
*B Shneiderman seminar on //~High-Impact Research: Blending Science, Engineering, and Design//
**DTUI 5ed
**[[Crossing the Chasm|http://www.amazon.com/Crossing-Chasm-Marketing-High-Tech-Mainstream/dp/0060517123]] by Jeffrey Moore
**[[The Rise of the Creative Class|http://www.amazon.ca/The-Rise-Creative-Class-Transforming/dp/1469281422]] by R. Florida
**Peacetech Conference
**[[engineeringchallenges.org|http://engineeringchallenges.org/]]
**[[Longitude Prize 2104|http://www.nesta.org.uk/project/longitude-prize-2014]]
**[[Innocentive|http://www.innocentive.com/]]
**[[Endless Frontier|http://www.amazon.ca/Endless-Frontier-Vannevar-Engineer-American/dp/0262740222]] by G. P. Zachary, bio of V. Bush, author of [[Science the Endless Frontier|https://www.nsf.gov/od/lpa/nsf50/vbush1945.htm]]
**[[Toward an Ecological Model of Research and Development|http://www.theatlantic.com/technology/archive/2013/04/toward-an-ecological-model-of-research-and-development/275187/]] by B. Shneiderman
**Donald Stokes & Pasteur's Quadrant, "Use Inspired Basic Research, Shneiderman's "~Theory-Inspired Applied Research"
**[[The Idea Factory|http://www.amazon.ca/The-Idea-Factory-American-Innovation/dp/B00B9ZBK9U]] by Jon Gertner (History of Bell Labs)
**Julia Lane (NSF) and [[StarMetrics|https://www.starmetrics.nih.gov/Star/News]]: [[Let's make science metrics more scientific|http://www.nature.com/nature/journal/v464/n7288/full/464488a.html]]
**High Impact Research Strategies:
***Choose actionable problems that address civic, business & global priorities
***Blend science, engineering, and design knowledge & research methods
***Seek interdisciplinary collaborations with diverse individuals & organizations
***Build on generalizable theories, principles & guidelines
***Develop prototypes that are tested with ever more realistic interventions
***Use quantitative big data & qualitative case study research methods
***Promote adoption & measure impact
*MUX CPSC 544 grad buddy paper selection
!!24 September
*CPSC 547 guest lecture by MB on //Perception, Cognition, and Effectiveness of Visualizations with Applications in Science and Engineering//
*CPSC 547 guest lecture on vis design resources
*no MUX
*meeting w/ MB re: collaboration, design studies, ~PhD retrospective project
*group website updates
!!25 September
*Pulse: reviewing annotation email thread
*reading: [[A Big Article About Wee Things|http://www.propublica.org/nerds/item/a-big-article-about-wee-things]] by [[Lena Groeger|http://www.propublica.org/site/author/lena_groeger]], ~ProPublica
*LUNCH
*Overview talk prep
*//Investigation of a tuberculosis outbreak in a BC homeless shelter: from bunk bed maps to bacterial genomes to Bayesian phylogenetics// - ~VanBUG talk by J. Gardy, 675 West 10th Avenue, Gordon and Leslie Diamond Family Theatre, BC Cancer Agency
*reviewing <<cite McKenna2014 bibliography:Bibliography>>
!!26 September
*~InfoVis group meeting: discussing <<cite McKenna2014>>
*Overview talk prep
*meeting w/ TM: [[minutes|TM-14.09.26]]
!References
<<bibliography>>
!!29 September
*Overview talk slides prep
!!30 September
*Overview talk slides prep
*Tuesday tea
*MUX CPSC 544 grad buddy paper reading: <<cite Schmidt2013a bibliography:Bibliography>>
!!01 October
*Overview talk script prep
*UBC school of music noon-hour concert
*CPSC 547 guest Q&A session on vis design resources
*MUX: CPSC 544 paper discussion x2
!!02 October
*BELIV copyright camera ready pdfs
*Overview talk script prep
*LUNCH
!!03 October
*~InfoVis group meeting: Overview practice talk
*catching up on Pulse emails re: annotation
*--meeting w/ TM: minutes--
!References
<<bibliography>>
!!06 October
*(flights from YVR tp YYZ to YOW)
*Overview talk revisions
*resolving BELIV embedded fonts issue
!!07 October
*(at cottage)
*Overview talk slides revisions
*VIS DC talk slides revisions
*emailing NHR from MSR re: internships
*--LUNCH scrum--
!!08 October
*(at cottage)
*VIS DC talk slides revisions
*BELIV DR Vis Tasks talk prep
*--MUX (PE,DC)-- (absent)
!!09 October
*(at cottage)
*Overview talk recording
*VIS DC talk recording
*BELIV PDE talk prep
*LUNCH
!!10 October
*(at cottage)
*reading <<cite Boy2014 bibliography:Bibliography>> for ~InfoVis group discussion
*~InfoVis group meeting: SHK paper discussion re: <<cite Boy2014>>
*Overview talk revisions based on JS feedback
*meeting w/ TM: [[minutes|TM-14.10.10]]
!References
<<bibliography>>
!!13 October
*(Thanksgiving)
!!14 October
*(train to YYZ)
*CHI review request / skimming CHI submission
*added [[R Graph Catalog|http://shinyapps.stat.ubc.ca/r-graph-catalog/#]] and [[OpenCPU|https://www.opencpu.org/]] to the [[vis design resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]]
*definitive BELIV talk format announced: 5 min + 20 min discussion
*responding to MS re: DDJ ~PhD position / forwarding
*2014-2014 GRAC committee request
*--LUNCH scrum--
!!15 October
*(vacation - YYZ)
*--MUX (544)-- (absent)
!!16 October
*(vacation - Mississauga / Windsor)
*--LUNCH--
!!17 October
*(vacation - Windsor / Toronto)
!References
<<bibliography>>
!!20 October
*(in Toronto, flight to YVR at 8:15pm ET)
*booked Europe trains / Airbnbs
*VIS DC email re: poster format / proceedings format TBA
*signed up for VIS Compass Blind Lunch on Tue Nov 11
*poster formatting
!!21 October
*[[JS Vis online book|http://jsdatav.is/intro.html]] via PB
*RISE renewal
*LUNCH scrum 10am (joining remotely - UPS delivery expected) - rail tickets delivered
*[[References: BELIV]] updated with BELIV'14 papers
*BELIV talk prep / rehearsal
*[[poster revision|http://goo.gl/jZobRS]] sent to TM
!!22 October
*reading BELIV papers appearing in same sessions: <<cite Rind2014 bibliography:Bibliography>>, <<cite Winters2014>>, <<cite Correll2014>>
*ICICS external review 11:50-12:30
*MUX: BELIV practice talks 2pm
*debrief w/ TM
*upgrading to OSX 10.10 Yosemite
!!23 October
*VIS DC format announced: 20 minutes + 20 minutes discussion
*upgrading Keynote, creating flattened versions of keynote presentations (packages with previous versions of keynote)
*talk revisions, scheduling BELIV PDE practice talk via Skype w/ co-authors (week of 14.10.27)
*LUNCH 11:30, feedback on poster from SHK, MN
!!24 October
*--~InfoVis group meeting--
*adding new CTOC pub to [[pubs page|http://cs.ubc.ca/~brehmer/publications.shtml]]
*VIS DC poster updates based on JM feedback
*[[PDE|http://goo.gl/PR22UA]] / [[DR BELIV|http://goo.gl/7mHmZZ]] talk prep / recording
*reading more BELIV papers appearing in same sessions: <<cite Kim2014>>, <<cite Scholtz2014>>
!!25-26 October
*worked on CHI review
*VIS DC poster updates based on JM, AP feedback
!References
<<bibliography>>
!!27 October
*website updates
!!28 October
*~DRVisTasks talk feedback from MS, PDE talk feedback from MT
*[[website updates done|http://cs.ubc.ca/~brehmer/]] sent out for feedback
*LUNCH scrum
*OS research chat
*checking out [[POP|https://popapp.in/]] via rapid prototyping roundup on [[usability geek|http://usabilitygeek.com/effective-techniques-rapid-prototyping/]]
*Tuesday Tea
*reading <<cite Kaptelinin2012 bibliography:Bibliography>>
!!29 October
*--GRAC meeting-- (GATS training already done)
*Pulse: EMU contact email
*additional ~DRVisTasks talk feedback from MS
*MUX: discussing <<cite Kaptelinin2012>>, //Brainport//
*CHI review
!!30 October
*formatting updates to poster
*LUNCH
*dept data science working meeting x836 12:30 - 1:50
*updated [[cv|http://cs.ubc.ca/~brehmer/mb_cv.pdf]], created [[resume|http://cs.ubc.ca/~brehmer/mb_resume.pdf]]
!!31 October
*~InfoVis group meeting: JF project update, SHK BELIV practice talk
*tweaks to poster, slides, [[cv|http://cs.ubc.ca/~brehmer/mb_cv.pdf]], created [[resume|http://cs.ubc.ca/~brehmer/mb_resume.pdf]], [[website|http://cs.ubc.ca/~brehmer/]], Pulse image cleaning / anonymization
*poster printing / travel document printing
!References
<<bibliography>>
!!03 November
*VIS talk rehearsals
*collecitng poster tube from MUX
*meeting w/ TM: [[minutes|TM-14.11.03]]
*adding [[BELIV discussion points|https://docs.google.com/document/d/1LQUNYcQ7bgPrxI_0MeMDSt-3QQSO1Vwepj-1a_E1gH4/edit?usp=sharing]]
!!04 November
*(municipal election advance voting)
*(haircut, currency exchange)
*LUNCH scrum (remote)
*~InfoVis group meeting: NM VAST practice talk 2pm
*meeting w KZ re: senior recruitment 3:30
*VIS talk rehearsals
*(packing for Paris)
!!05 November
*(depart for Paris)
*reading BELIV papers / adding discussion points
*reading <<cite Kaastra2014 bibliography:Bibliography>>, <<cite Aupetit2014>>, <<cite Brehmer2014a>>, <<cite Brehmer2014b>>
*editing CHI review
!!06 November
*(arrival in Paris)
*VIS DC talk rehearsal
!!07 November
*(in Paris)
*VIS DC talk rehearsal
!!08 November
*VIS DC
!!To do:
*revise CHI review (due Nov 10)
*read Overview RW appearing @ VIS 14: <<cite Cui2014>>, <<cite Alexander2014>>
Later:
*publish ggplot implementation of ~LineUp as a gist / blog post / Shiny App (later)
*read [[Geer on cybersecurity|http://geer.tinho.net/geer.blackhat.6viii14.txt]]
!References
<<bibliography>>
!!09 November
*VIS talk rehearsals
*VIS 2014 / Tutorial on design studies
!!10 November
*~VIS2014 / BELIV 2014
*CHI review due
!!11 November
*~VIS2014
*Austrian party
!!12 November
*~VIS2014, poster presentation
*West Coast Party
!!13 November
*~VIS2014
!!14 November
*~VIS2014, Overview presentation
!!To do:
Later:
*publish ggplot implementation of ~LineUp as a gist / blog post / Shiny App (later)
*read [[Geer on cybersecurity|http://geer.tinho.net/geer.blackhat.6viii14.txt]]
!References
<<bibliography>>
!!24 November
*(travel home from DUS/LHR)
!!25 November
*LUNCH scrum
*updates to ~InfoVis group news page
*preparing expense reimbursement claims
*reviewing slides from [[Everything except the chart - IEEE VIS 2014 Tutorial|http://dominikus.github.io/webvis-tutorial/www/#/]]
*emailing NHR, BL, BCK re: MSR / computational journalism, internship logistics
*Tuesday tea
!!26 November
*--GRAC meeting canceled--
*updates to ~InfoVis group resources page
*BCK (Konstanz) responded with manuscript re: comp. journalism
*VIS DC feedback audio transcription
*reading 544 paper for MUX on SNS for older adults
*floor warden meeting
*MUX 544 reading
!!27 November
*CHI rebuttal and review revision
*VIS DC feedback audio transcription
*scientific communication through visualization survey
*VIS Roundup notes
*LUNCH meeting
!!28 November
*~InfoVis group meeting: VIS roundup
*checking out [[vislists|http://vislists.anandsainath.com/list/]]
*UDLS
!!To do:
Later:
*reviewing [[Black Box on Interaction|http://vcg.informatik.uni-rostock.de/~hs162/blackbox/tutorial.html]] tutorial from VIS 2014
*publish ggplot implementation of ~LineUp as a gist / blog post / Shiny App (later)
*read [[Geer on cybersecurity|http://geer.tinho.net/geer.blackhat.6viii14.txt]]
!References
<<bibliography>>
!!01 December
*~InfoVis group meeting: BR, GM, TM ~EuroVis paper draft review
*reviewing updates to Pulse EMU, external review meeting scheduled for next Tuesday (Dec 9)
*MUX, ~InfoVis twiki updates
*GATS login, created HCI list
*checked out [[FriendsInSpace|http://app.friendsinspace.org/]]
*meeting w/ TM 3:30: [[minutes|TM-14.12.01]]
!!02 December
*LUNCH scrum
*Pulse: touching base w/ AL, JN, KT
**[[Pulse acquired by EnerNOC|http://www.pulseenergy.com/pulse/a-letter-from-our-co-founder-and-ceo-david-helliwell/]]
*reading [[LC's advice re: vis consulting|http://blogger.ghostweather.com/2013/11/data-vis-consulting-advice-for-newbies.html]]
*reading DB + MS' [[Everything except the chart|http://dominikus.github.io/webvis-tutorial/www/]] tutorial slides
**[[Hipster Stack|http://leemart.in/hipster-stack]], [[Sass|http://sass-lang.com/install]]
*career chat w/ JM
*reading BHK+BL manuscript
*[[ShinyFork|https://mattbrehmer.shinyapps.io/ShinyFork/]] maintenance
*reading [[p5.js, node.js, socket.io|https://github.com/lmccart/p5.js/wiki/p5.js,-node.js,-socket.io]]
!!03 December
*--GRAC meeting canceled--
*(bike repair)
*finished reading BHK+BL manuscript
*[[p5.js|http://p5js.org/reference/#/p5/set]] experimentation: Processing scatterplot maps
*checking out [[tryGit|https://try.github.io/]], [[explain shell|http://explainshell.com/]], [[how to learn javascript properly|http://javascriptissexy.com/how-to-learn-javascript-properly/]], [[NYU ITP Creative JavaScript Fall 2014 course|https://github.com/lmccart/itp-creative-js]],
*meeting w/ JF + TM re: timeline vis
!!04 December
*LUNCH meeting (㊍)
*[[p5.js|http://p5js.org/reference/#/p5/set]] experimentation: [[Processing scatterplot maps|http://bl.ocks.org/mattbrehmer/a129bf7a88f83ef1d0d5]]
!!05 December
*trying out [[bertifier.com|http://bertifier.com/]]
*revising JF's journalist interview script re: timelines
*responding to BCK email re: ~VisJockey
*no ~InfoVis meeting (~EuroVis deadline)
!References
<<bibliography>>
!!08 December
*Cosmin Munteanu (U Toronto Mississauga): //Breaking the barriers of conventional interfaces: How multimodal interfaces and new media can make interaction with assistive technologies more natural.//
*Lunch w/ CM
*Pulse: Skype call w/ JN, AL to review interview protocol
*data scraping for aggregate annual album review ranking visualization, discovered [[AOTY|http://www.albumoftheyear.org/]]
*[[UBC Journo PK featured in the Saturday G&M|http://www.theglobeandmail.com/news/british-columbia/ubc-journalism-director-looks-to-philanthropy-to-fund-reporting-centre/article21980911/]]
*response from BCK re: ~VisJockey
*reading [[Am I a data scientist?|http://robjhyndman.com/hyndsight/am-i-a-data-scientist/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+Hyndsight+%28Hyndsight%29]] by RJ Hyndman
*checking out [[Tools and Libraries for Building Web-based Data Visualisations|http://charts.animateddata.co.uk/datavistree/]] by P Cook
*PB's CPSC 547 project [[featured on NY's FlowingData blog|http://flowingdata.com/2014/12/08/detailed-visualization-of-nba-shot-selection/]]
!!09 December
*--LUNCH scrum--
*Pulse: visiting EM user MT at SSD, interviewing facilities manager RS at LTC, meeting w/ KT @ Pulse
*checking out [[Keshif|http://keshif.me/]] by MA Yal穮 (UMD)
*checking out [[Statmuse|https://www.statmuse.com/beta]]
!!10 December
*IEEE VIS surveys
*HCI@UBC: //#~MOAinteractive: Designing the Museum Experience//
*Pulse: editing notes from yesterday's user interview sessions
*revising JF interview questions for journos
*meeting w/ JF + TM, response from PK re: [[global torture timeline|http://globalreportingcentre.org/torture-timeline/]]
*checking out [[lineup.js|http://caleydo.github.io/lineup.js/demo/#]] by JKU's SG
*parsing [[QS University Rankings 2012-2014|http://www.topuniversities.com/qs-world-university-rankings]], preparing rank benchmark dataset
!11 December
*parsing [[AOTY'S Complete Music Rankings of 2014|http://www.albumoftheyear.org/ratings/overall/2014/]] using Sublime + regex, scraping metadata from linked album pages in R, preparing rank benchmark dataset
*LUNCH meeting (㊍)
*UBC CS Dept. Holiday party
*response from Sandia's LM re: IEEE VIS tutorial design study slides
!!12 December
*parsing [[AOTY'S Complete Music Rankings of 2014|http://www.albumoftheyear.org/ratings/overall/2014/]] using R
*CPSC 547 final presentations
*meeting w/ JG, AC, PB re: CDC vis
*email from Oculus's RB re: job search
*~EuroVis review request
!References
<<bibliography>>
!!15 December
*responding to ~EuroVis review request
*responding to RB (Oculus)
*Pulse: completing notes from EMU interviews, sending to JN, AL
*meeting w/ TM: [[minutes|TM-14.12.15]]
*ranking vis: complete data parsing scripts, create github repository
!!16 December
*LUNCH scrum
!!17 December
*~SoundConsensus (ranking vis)
*--meeting w/ JF + TM re timeline vis project--
!!18 December
*~SoundConsensus
*LUNCH meeting
!!19 December
*~SoundConsensus
*CSGSA AGM, Holiday Party
!!22 December
*(holiday)
!!23 December
*~SoundConsensus (ranking vis)
!!24 December
*~SoundConsensus (ranking vis)
!!25 December
*(holiday)
*LUNCH meeting
!!26 December
*(holiday)
!!To do:
*read FA+NHR+BL paper submission (when available)
!!29 December
*~EuroVis review requests
*emailing MB re: work domain analysis
*emailing AY (re: Keshif version of typology)
*released [[SoundConsensus|http://mattbrehmer.github.io/SoundConsensus/]]
*updated [[group resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]]
!!30 December
*adapted ~SoundConsensus code into [[UniversityRanks|http://people.cs.ubc.ca/~brehmer/demo/universities/]]
!!31 December
*(holiday)
!!01 January
*(holiday)
!!02 January
*(holiday)
!!05 January
*(at home, ill)
*Pulse: [[Portfolio Sandbox|https://mattbrehmer.shinyapps.io/PortfolioSandbox/]] (login: ieeevis / vis2015), created [[github repo|https://github.com/mattbrehmer/PortfolioSandbox]]
*Pulse pre-paper talk
*GRAC reviews
!!06 January
*LUNCH scrum 11:30
*Pulse pre-paper talk
*GRAC reviews
!!07 January
*--MUX 2pm--
*Pulse pre-paper talk
*GRAC reviews
!!08 January
*LUNCH meeting 11:30
*Pulse pre-paper talk
*meeting w/ PR re: global housing vis
!!09 January
*Pulse pre-paper talk
*~InfoVis group meeting 12:30: ~MBo Glue pre-paper talk
*meeting w/ JF + TM re: timeline vis
*meeting w/ TM: [[minutes|TM-15.01.09]]
!!12 January
*Pulse pre-paper talk v2
*GRAC reviews
*GRAC meeting 1pm
!!13 January
*GRAC reviews
*LUNCH scrum 11:30
*meeting w/ ~MBo 12:30
*Pulse pre-paper talk v2
*reading FA+NHR+BL paper submission
!!14 January
*network disk quota exceeded, cleanup required
*[[freebusy calendar|http://www.cs.ubc.ca/~brehmer/freebusy.shtml]] added to website
*GRAC reviews
*HCI @ UBC
*--MSR skype call w/ BL, NHR-- postponed until 20 January
*~EuroVis reviewing complete
!!15 January
*GRAC reviews
*LUNCH meeting 11:30
*Pulse pre-paper talk v2
*meeting w/ PR, JF re: timeline vis project
*DLS: //Learning to Make Better Decisions: Challenges and Opportunities for the 21st century// by Csaba Szepesvari
!!16 January
*Pulse pre-paper talk v2
*feedback to MB re: talk abstraction for GLUE paper
!!19 January
*~EuroVis review proofreading; responding to PR re: additional review request
*GRAC secondary reviews reminders
*Pulse pre-paper talk v2
*meeting w/ TM: [[minutes|TM-15.01.19]]
!!20 January
*MSR skype call w/ BL, NHR
*LUNCH scrum 11:30 (via Skype)
*GRAC review
*Pulse pre-paper talk v3
!!21 January
*MUX
*meeting w/ TM + JF re: timeline vis
*GRAC offers initiated x2
*Pulse pre-paper talk v3
!!22 January
*meeting w/ TM re: Pulse pre-paper talk v4
*Pulse pre-paper talk v4
*LUNCH meeting 11:30
*emailing JS re: documents for timelines
*meeting w/ PK (UBC Journo), TM, JF re: timeline vis project
!!23 January
*GRAC review
*~InfoVis group meeting: MB Pulse pre-paper talk
*meeting w/ TM: pre-paper debrief
!!26 January
*GRAC reviews
*GRAC meeting
*Pulse pre-paper talk v5
*timelines: meeting w/ JF: questions for UBC journalism students re: timelines
!!27 January
*GRAC reviews
*LUNCH scrum 11:30
*Pulse pre-paper talk v5
!!28 January
*dentist appt 8:30am
*[[RK Tableau reddit AMA|http://www.reddit.com/r/tableau/comments/2twgih/iama_visual_analysis_researcher_at_tableau_robert/]]
*Pulse pre-paper talk v5
*timelines: timing test
!!29 January
*Pulse pre-paper talk v5
*LUNCH meeting 11:30
!!30 January
*Google ~PhD luncheon
*--~InfoVis group meeting: TM practice talk 2pm (via Skype)-- (canceled)
!!02 February
*GRAC HCI organizing
*timelines: meeting w/ JF: pre-pre-paper talk
*Pulse pre-paper talk v5
*MSR onboarding document prep
!!03 February
*LUNCH scrum 11:30
*Pulse pre-paper talk v5
*NYU guest seminar / demo request from EB
!!04 February
*submitted MSR onboarding documentation
*Pulse pre-paper talk v5
*MUX 2pm: research updates
*timelines: meeting w/ JF: pre-pre-paper talk 3pm
!!05 February
*meeting w/ TM + JF: timeline pre-pre-paper talk 11am
*LUNCH meeting 11:30
*Pulse pre-paper talk v5 at Pulse 3pm
!!06 February
*pick up transcript
*meeting w/ TM + JF: timeline pre-pre-paper talk 11am
*--~InfoVis group meeting: NM pre-paper talk 12:30 pm-- (canceled)
*IMAGER SIGGRAPH showcase
*meeting w/ TM + JF: timeline pre-pre-paper talk (cont.) 4:30 pm
!!09 February
*(family day holiday)
*Pulse: transfering pre-paper speaker notes to IEEE conference paper format (5 pages + refs)
!!10 February
*responding to FCV folks re: UX research job
*LUNCH scrum 11:30
*TLC: think-aloud / speed test take 2
*TLC: meeting w/ Codi (UBC Journalism)
*MS ~PowerBI demo (Henry Angus 969) 5pm
!!11 February
*GRAC HCI coordination
*GRAC meeting 11am
*HCI@UBC 12pm - wearables@UBC
*--~InfoVis group meeting 1:30pm - if need be?-- (canceled)
*MSR onboarding documents
*--TLC: JF meeting w/ PR--
*--TLC: skype call w/ Carys--
*TLC: meeting w/ TM + JF
!!12 February
*GRAC reviews / admin
*responding to Google recruiter email
*immigration docs for MSR
*LUNCH meeting 11:30
*RSVP'ed for Hacks/Hackers meetup
*Pulse: paper writing: tables and figures
!!13 February
*phone call w/ FCV folks re: UX jobs
*~InfoVis group meeting: JF timelines pre-paper talk 12:30 pm
*meeting w/ TM + JF: timeline pre-paper talk debrief
!!16 February
*GRAC admin
*review request from CT re: vis lit + older adults
*Pulse: paper writing
*TM meeting 2:30: [[minutes|TM-15.02.16]]
*--~InfoVis group meeting: DA/TM pre-paper talk 12:30pm-- (delayed to 15.02.23)
!!17 February
*Pulse: paper writing
*TLC: creating project page
!!18 February
*Pulse: paper writing
*~InfoVis group meeting: NM pre-paper talk 10am
*TLC: meeting w/ TM + JF 2:pm
!!19 February
*Pulse: paper writing
!!20 February
*Pulse: paper writing
!References
<<bibliography>>
!!23 February
*GRAC review
*GRAC meeting
*update re: VIS DC reimbursement
*responding to RB / Uncharted
*scheduling annual committee meeting
*Pulse: [[first draft complete|https://www.sharelatex.com/project/54d55c089b8eb8a228028597]] / edit token to TM
*TM meeting 2:30: Pulse paper / [[minutes|TM-15.02.23]]
!!24 February
*LUNCH 11:30
*~InfoVis group meeting: EH pre-paper talk 12pm
*grad seminar on finding a job
!!25 February
*TLC: meeting w/ TM + JF 4pm
!!26 February
*LUNCH scrum 11:30
*Hacks/Hackers pub night 6:30 @ Moose's Down Under (830 Pender)
!!27 February
*--~InfoVis group meeting 12:30 (open)--
*meeting w/ TM re: Pulse paper
!!02 March
*TLC paper writing
*TLC benchmarking
*Skype meeting w/ prospective grad 4:30
*IVJ review requests (x2)
!!03 March
*TLC paper writing
*TLC benchmarking
*LUNCH 11:30
*initiating PCS submission for Pulse paper
*meeting w/ prospective grad 7pm
!!04 March
*TLC paper writing
*TLC benchmarking
*MUX 2pm
*Pulse paper token return check-in / inquiries re: data, # users
!!05 March
*TLC paper writing
*meeting w/ PS (Tyee, Hacks/Hackers) 6pm
!!06 March
*TLC paper writing
*prospective grad visit day / demos / lunch
*CSGSA coffee house
!!09 March
*listening data stories #47
*SE recruiting talk
*--GRAC meeting-- postponed until next week
*Pulse EMU paper token JN -> MB
*TLC paper token: JF
*meeting w/ TM: [[minutes|TM-15.03.09]]
!!10 March
*Pulse EMU paper editing / video creation
*LUNCH 11:30
!!11 March
*Pulse EMU paper sent to ~InfoVis group
*HCI@UBC: Kori Inkpen (MSR) re: video interaction
*MUX group chat w/ KI
*meeting w/ JF + TM 3pm
*5:30 Maria Klawe talk re: women in STEM
!!12 March
*10am Brian Bailey talk ICICS 206
*chat w/ JF + Brian Bailey 11:50 in x660
*TLC paper writing / editing
*Pulse EMU video creation
*meeting w/ JF + TM 3pm
!!13 March
*~InfoVis group meeting: MB paper draft
*debrief w/ TM 4:30pm
!!To do:
*IVJ BELIV special issue reviews x2
*timeline vis lit review for MSR
!!16 March
*HCI GRAC admin
*GRAC meeting: general admits
*coordinating prospective grad KDR visit
*Pulse EMU paper revisions
*CSCW HCI@UBC SFU SIAT joint reception 7-10pm
!!17 March
*Pulse EMU paper revisions
*--LUNCH scrum-- canceled (JM at CSCW)
!!18 March
*~InfoVis group meeting: EH paper draft
*--MUX-- canceled due to CSCW
*meeting w/ TM + JF 2pm
*TLC benchmarking?
*Pulse EMU paper revisions: token to TM
!!19 March
*Pulse EMU paper revisions
*--LUNCH-- canceled (faculty recruiting)
*~InfoVis group meeting: AC practice talk
*talk to Joyce re: 4YF funding interruption
!!20 March
*prospective grad KDR visit
*~InfoVis group meeting: JF paper draft
*debrief w/ TM 2pm
*Pulse: meeting w/ TM 5pm
!!23 March
*Pulse EMU paper revisions: token to KT, JN
*re: MSR / BL, NHR re: Apr 2 visit
*TLC edits
*IVJ reviews
!!24 March
*TLC edits
*Pulse: supplemental video
*IVJ reviews
*LUNCH
!!25 March
*TLC edits
*Pulse: supplemental video
*IVJ reviews
*meeting w/ TM + JF 4pm
*industry panel 5:30
!!26 March
*TLC edits
*Pulse: supplemental video
*LUNCH scrum
*doctor's appt 3:15
*--SE faculty recruiting: helping developers find useful tools--
!!27 March
*TLC edits
*--~InfoVis group meeting: MB paper draft 12:30pm-- (canceled)
!!30 March
*Pulse EMU paper + video revisions
*SE faculty candidate talk
*meeting w/ TM: minutes
!!31 March
*--LUNCH meeting-- (deadline)
*--grad seminar on web dev--
*Pulse EMU paper + video revisions
*~InfoVis deadline 5pm
!!25 April
*volunteer lunch 12pm
*MUX: FE + KZ research updates 2pm
*IVJ reviews
!!26 April
*meeting w/ BL + NHR from MSR: 10-12:30
*timeline vis lit review for MSR
*--LUNCH scrum--
*IVJ reviews
!!27 April
*(Good Friday holiday)
*IVJ reviews submitted
!!06 April
*(Easter Monday)
*responding to IEEE VIS review requests
*reimbursement of ~ShareLaTeX and Evernote
*official vs. unofficial leave inquiry; internship travel request made to student health plan
!!07 April
*(departing for Japan)
*IEEE VIS reviews
!!08 April
*(Japan)
!!09 April
*(Japan)
!!10 April
*(Japan)
!!27 April
*transferring [[CHI2015]] notes
*GRAC lunch meeting 1pm
*(went home, sick)
!!28 April
*--LUNCH scrum 11:30-- (at home, sick)
*updating [[CHI2015]] notes w/ links to papers
*Annual ~PhD progress report
*VIS reviews
!!29 April
*(at home, sick)
*MSR TL brainstorming
*VIS reviews
*--meeting w/ TM 2:30-- (postponed, sick)
!!30 April
*Annual ~PhD progress report slides
*LUNCH meeting 11:30
*MSR TL brainstorming
*VIS reviews
!!01 May
*~InfoVis group meeting: summer meeting schedule, CHI recap
*MSR TL brainstorming
*VIS reviews
*meeting w/ TM 2:00
!!04 May
*MSR meeting slides prep
!!05 May
*MSR meeting slides prep
*LUNCH scrum
*VIS reviews
!!06 May
*meeting at MSR w/ TM, BL, NHR, BB
!!07 May
*--LUNCH meeting-- (returning from MSR)
*supervisory meeting 3:30 ICCS 304
*VIS reviews
!!09 May
*VIS reviews
!!10 May
*VIS reviews
!!11 May
*~InfoVis group scheduling
*~InfoVis group meeting: RM graph-drawing pre-paper talk
*JF thesis review
!!12 May
*~EuroVis program review
*LUNCH scrum
*VIS review discussion
*timeline project literature search + insight characterization; dataset search
!!13 May
*JF thesis abstract review
*VIS review discussion
*timeline project literature search + insight characterization; dataset search
!!14 May
*--~InfoVis group meeting-- (canceled, TM in California)
*LUNCH meeting 11:30
*timeline project literature search + insight characterization; dataset search
!!15 May
*timeline project literature search + insight characterization; dataset search
!!18 May
*(Victoria Day holiday)
*VDV talk prep
!!19 May
*VDV talk prep
*LUNCH meeting: AP GI practice talk
*VIS review discussion
*timeline project literature search + insight characterization; dataset search
*[[VDV meetup: Come discuss the future of data and design!|http://www.meetup.com/Vancouver-Data-Visualization/events/222352638/]]
!!20 May
*timeline project literature search + insight characterization; dataset search
*reading [[Boy et al 2014|http://dl.acm.org/citation.cfm?id=2702452]] for ~InfoVis group meeting
*MUX: Prototyping tools
*learning [[angular.js|http://campus.codeschool.com/courses/shaping-up-with-angular-js]]
!!21 May
*~InfoVis group meeting: NM leads paper discussion
*--LUNCH meeting 11:30-- (conflict w/ ~InfoVis group meeting)
*timeline project literature search + insight characterization; dataset search
**BB email discussion re: timeline distortion
*learning [[angular.js|http://campus.codeschool.com/courses/shaping-up-with-angular-js]]
*meeting w/ TM: minutes
!!22 May
*responding to recruiter email
*BB email discussion re: timeline distortion
*learning [[angular.js|http://campus.codeschool.com/courses/shaping-up-with-angular-js]]
*timeline project literature search + insight characterization; dataset search
*meeting re: vis / edu tech / craft beer - 6pm @ Barney's on Main
!!25 May
*Data Science candidate research talk 11am
*joining ~SeaVis meetup / google groups lists
*timeline project literature search + insight characterization; dataset search
*reading //Cartographies of Time//
!!26 May
*--Data Science candidate vision talk 10am--
*Codeschool tutorials on Chrome devtools
*LUNCH meeting: MH GI practice talk
*reading about Angular.js / react.js, combining w/ d3.js
*timeline project literature search + insight characterization; dataset search
!!27 May
*timeline project literature search + insight characterization; dataset search
*meeting w/ TM 2:30: minutes
!!28 May
*--UBC CS Dept. Retreat Day 1-- (did not attend)
*call w/ headhunter / recruiter for data visualization design position
*[[angular.js tutorial on codecademy|http://www.codecademy.com/learn/learn-angularjs]]
*WTF: [[Visualizing techniques with plants for Interaction Design|http://blog.derhess.de/2014/06/02/visualizing-techniques-with-plants-for-interaction-design/]] (June 2, 2014)
*[[Keshif tasks demo|http://keshif.me/demo/vis_tasks.html]]
*--~InfoVis meeting-- canceled
*--LUNCH meeting-- canceled
!!29 May
*(flying to New Orleans)
*timeline project literature search + insight characterization; dataset search
*timeline biography sketching
*reading paper draft for ~InfoVis group meeting 06/01
!!01 June
*(in New Orleans)
!!02 June
*(returning from New Orleans)
*reading paper draft for ~InfoVis group meeting 06/01
!!03 June
*commenting on paper draft for RM, TM via email
*timeline bio for MSR
!!04 June
*updating group website w/ TM's Vivid Sydney talk
*meeting w/ ZB (MIT CSAIL)
*SN ~PhD thesis defense practice talk
*timeline bio for MSR
*--~InfoVis meeting-- canceled (TM away)
*--LUNCH meeting-- canceled (JM away)
!!05 June
*preparing to depart for MSR
Some helpful links:
*http://cs.ubc.ca/~brehmer (home page)
*http://matthewbrehmer.net/ (personal non-academic site)
*My [[CV|http://people.cs.ubc.ca/~brehmer/mb_cv.pdf]] / [[linkedIn|http://ca.linkedin.com/pub/matthew-brehmer/26/692/98b]] profile
*twitter: [[@mattbrehmer|http://twitter.com/mattbrehmer]]
*[[M.Sc research|https://bugs.cs.ubc.ca/cgi-bin/twiki/view/Main/C-TOC]]
*[[Tamara's home page|http://www.cs.ubc.ca/~tmm/]]
*[[Joanna's home page|http://www.cs.ubc.ca/~joanna/]]
*[[InfoVis group|https://bugs.cs.ubc.ca/cgi-bin/twiki/view/Imager/InfoVisGroup]]
*[[Multimodal User Experience Group|http://www.cs.ubc.ca/nest/imager/imager-web/Meetings/mux.html]]
*[[IMAGER Lab for Visualization, Graphics, & HCT|http://www.cs.ubc.ca/labs/imager/imager.php]]
*[[Department of Computer Science|http://www.cs.ubc.ca/]]
Here is a list of topics contained in this literature review:
<<list filter [tag[litReview]]>>
Multi-dimensional In-depth Long-term Case Studies (<<cite Shneiderman2006 bibliography:Bibliography>>).
*''multi-dimensional'': interviews, surveys, automated logging - assess performance, interface efficacy and utility
*''in-depth'': intense engagement with expert users, to the point of becoming a partner or assistant
*''long-term'':longitudinal studies, from training through to proficient usage of a tool, leading to changes in strategy for expert users
*''case studies'': detailed reporting about a small number of domain experts working on their own problems in their normal setting.
MILCs address the situated nature of most work activity, and are inspired by [[Action Research]], which aims to find ways to change or improve processes over and above the pure ethnographic approach (studying processes for their own sake). MILCs build on case of field studies in that they take place over weeks and months, which is necessary to fully understand the processes studied.
*overview of artifacts / data collected so far
**notes from interviews
**audio from interviews
**participant documentation and other artifacts
**participant publications and literature base
**related literature
**existing summaries
*ethics protocol - my name to be added to the existing application
*ethnographic / qualitative research references:
**[[Charmaz: Constructing Grounded Theory: A Practical Guide through Qualitative Analysis|http://www.amazon.com/Constructing-Grounded-Theory-Qualitative-Introducing/dp/0761973532]] (copy at UBC library: Temporarily shelved at KOERNER LIBRARY reserve collection (Floor 3) Call Number: H62 .C3595 2006)
**Fetterson book (Michael has physical copy)
*SVN permissions needed (GL: allow a couple of days)
*keynote/pages/numbers incompatibilities (resolved if .zip extension is added to filename, Mac Preview .pdf available to read)
**MS sent summary table in .xls
*to do: read notes, summaries, audio, transcripts
*prioritizing remaining case studies for summarization:
#HL - super interesting, not complex
#DH - super interesting, complex
#JW - interesting, difficult
#JS - still need more data
#A&S, JB
*to read DR refs:
**~DimStiller paper
**Tenebaum ref (nonlinear DR)
**Data-driven reflectance model paper (Matusik)
*additional interviews before Dec 2?
**MH, SI (defer to 2012), EK, K (ask TM next week)
*discussing laptop specs
*discussing summaries: JB, HL, AS, DH, JW, SNA
**remaining: JS (need info from TM, SI)
**AS: validating w/ Paraglide paper
**JB: initially seeking clusters as ends, later as means to discover semantics of new dimensions; same category as CM (patterns of boaters);
***CHI note accepted; validity of dimensions as input to PCA questionable
**model builders: SNA, AS, GM
*filling in the summary table, adjusting taxonomy: Jan 9 10am PST
*additional papers to validate / improve taxonomy (i.e. Matusik's BRDF paper), fuel cell use case from SB's Paraglide paper
[[Home]]
[[Journal]]
[[Meeting Minutes]]
[[Literature Review]]
[[References]]
[[Research Projects]]
[[Glossary]]
[[Dedoose|https://www.dedoose.com/App/?Version=4.2.82]]
^^ Courses
[[EPSE 595]]
[[CPSC 508]] ^^
^^ Apps
[[SoundConsensus|http://mattbrehmer.github.io/SoundConsensus/]]
[[UniversityRanks|http://people.cs.ubc.ca/~brehmer/demo/universities/]]
[[PortfolioViz Sandbox|https://mattbrehmer.shinyapps.io/PortfolioSandbox/]]
[[ShinyFork|https://mattbrehmer.shinyapps.io/ShinyFork/]]
[[Informed Omnivore|http://cs.ubc.ca/~brehmer/demo/omnivore/InformedOmnivore.html]] ^^
^^ [[spark.rstudio|http://spark.rstudio.com:8787/]] ^^
^^ [[shinyapps.io|https://www.shinyapps.io/dashboard]] ^^
^^ [[Links]]
[[BibTeX]]
[[Settings]] ^^
^^ [[TiddlyWiki|http://www.tiddlywiki.com/]] <<version>>
2014 [[M. Brehmer|http://cs.ubc.ca/~brehmer/]]
[[InfoVis Group|http://www.cs.ubc.ca/group/infovis/]]
[[public-access wiki|http://cs.ubc.ca/~brehmer/wiki/]]
[[@mattbrehmer|http://twitter.com/mattbrehmer]] ^^
!!!Conceptual Understanding
A subset of DR, and alternative to classical MDS. The process by which high dimensional data (HDD) is described in a low-dimensional basis, for the purposes of ''data compression'', ''de-noising'', and for ''visualization''. The process involves selection of a ''distance metric'' (linear or manifold). Curvier manifolds require denser data.
>//"[a] key tool in your object recognition toolbox, formal framework for many different ad-hoc object recognition techniques"//
*''ISOMAP'' is used when points have a complicated, non-linear relationship to one another. In these cases classical MDS and PCS break down. ISOMAP can handle such data. A 2D data set in a 3D space (i.e. the [[Swiss Roll]] surface), nearest-neighbours are computed using the geodesic distance (along the surface), rather than using Euclidean distance. Inter-point distances are preserved. It is sensitive to noise and noise edges, it preserves global structure, has few free parameters, but it is computationally expensive. The true dimensionality of the data can be estimated from the decrease in error as the dimensionality of the set of reduced dimensions is increased. The speed at which the algorithm converges depends on the parameters of the manifold (curvature, branch separation), and point density. The process:
#build a sparse graph with //k// nearest-neighbours
#infer other inter-point distances by finding shortest paths
#build a low-dimensional embedded space to best preserve a complete distance matrix, according to some error function
*[[Local Linear Embedding]] is a nonlinear approach that analyzes overlapping local neighborhoods in order to determine local structure.It preserves local topology but distorts global structure. It is incremental and fast, simple linear algebra, has one free parameter, and produces no local minima. However, they cannot represent global structure within a single coordinate system.
*Other Manifold Learning techniques:
**Laplacian eigenmaps
**Kernel PCA
**Kohonen Self-Organizing Maps (clustering method?)
!!!Comments/Questions:
*Q: Does this correspond to Euclidean vs. Geodesic?
**A: yes. often a point of confusion when assumptions are made regarding the sparsity of the manifold. Still not sure what the task is here.
*Q: Does Manifold Learning = DR? = MDS?
**A: No. Manifold learning is a subset of DR, an alternative to MDS
!!!Terminology I don't understand
*Lagrange multiplier
*reconstruction error (presumably error term in low-dimensional space)
!!!Sources:
Thompson, D. R. //Manifold learning with applications to object recognition//. Presentation for Advanced Perception.
!!Literature Notes & Commentary
<<cite Prown1982 bibliography:Bibliography-EPSE595>>
*Reminiscent of Don Norman's teapot collection, his //Psychopathology of Everyday Things//, emotion and design
*Prown's emphasis on historical examples, studying material culture from antiquity / past eras - can this apply to intangible artifacts of the recent past? i.e. is software subject to material culture? can we take a material culture approach to studying early GUI interfaces, to early visualization programs?
*Into the domain of semiotics, a revealing of intention - the semiology of graphics (Bertin)
*B. Buxton's CHI exhibit of interface technology over the years
<<cite Mathison2009>>
*Compliments Massironi's 2001 book //The Psychology of Graphic Images// - dimensions of ''verisimilitude'' and ''veridicality'', coherence, relevance, justifiability, contextuality
**orthogonal viewpoints, impossible vantage points - these still convey meaning, often easier to understand than verisimilar images of buildings, maps
*other PSYC 579 refs: Monmonier's //How to Lie with Maps//, ~MacEachren's //How Maps Work//, Kuiper's //"Map in the Head" Metaphor//, Golledge and Stimson's //Spatial Cognition//
*Often in visualization validation is offered in the form of images, proof of concepts, demonstrations of graphical rendering technique - often in a post-positivist framework - a validated hypothesis
*Data reporting / data journalism increasingly making use of compelling info-graphics, i.e. NYT's graphics dept., again a post-positivist framework
**The role of aesthetics in graphical displays - this can be compelling but at other times distracting if poorly applied
*Images still supplementary in qualitative research in HCI, in field work or in [[Grounded Theory|Grounded Evaluation]] studies
!!References
<<bibliography>>
Here are recorded meeting minutes:
!!TM:
<<list filter [tag[TMmeetings]]>>
!!JM:
<<list filter [tag[JMmeetings]]>>
A visualization of visualizations, organizing single visualizations into an organized form:
*A list
*''SPLOM'': a scatterplot matrix
*Other matrix layouts:
**Small multiples, Trellis, Lattice, Facets
*Iconic versions of visualizations in a scatter plot
!!!!Source:
<<cite Bertini2011 bibliography:Bibliography>>
<<cite Becker1996a>>
~Multi-Level Interfaces display hierarchically-organized task-dependent data at multiple visual levels.
Among [[Multi-Level Interfaces]], there exist [[Temporal Interfaces]] and [[Simultaneous Interfaces]], the former making use of a single display affording [[Panning]] and [[Zooming]], while the latter simultaneously presents [[Low-Level Displays]] and [[High-Level Displays]]. Among [[Simultaneous Interfaces]], there is further distinction between [[Separate Interfaces]] and [[Embedded Interfaces]]. The former, sometimes referred to as [[Overview + Detail Interfaces]], refers to 2 or more distinct displays presenting different levels of the data, often requiring view-coordination and coordinated interaction between displays (such as [[Linking]] or [[Brushing]]). The latter refers to a single display with a low-level context region and a high-level focus region, otherwise known as a [[Focus + Context Interfaces]]. This usually, but not always, involves a spatial distortion of the data in the high-level region surrounding the low-level region.
!![Lam2010]
[>img(20%, )[decision tree to create multi-level display|http://www.cs.ubc.ca/~tmm/swpix/mrmc-dtree.png]]
<<cite Lam2010 bibliography:Bibliography>> surveyed 22 research papers examining differences between [[Single-Level Interfaces]] and three categories of [[Multi-Level Interfaces]], resulting in their contribution of a decision tree for providing interface design recommendations. The decision tree (shown at right) encompasses several factors, including the interface elements, the interactions afforded by the interface, and how the data is represented in the interface (the type, amount, and transformations performed on the original data, its visual quality and quantity). The intended use of the data and the tasks of the user must also be considered, as well as the intrinsic organisation of the data.
Their interface design recommendations include the inclusion one view level per task-relevant data level. [[High-Level Displays]] should not contain too much or too little (or illegible) data. [[Simultaneous Interfaces]] are preferable when tasks require multi-level answers or single-level answers with multi-level clues. Otherwise [[Temporal Interfaces]] can be used and may outperform [[Simultaneous Interfaces]] when single-level answers are required with single-level clues.
Future work should aim to provide design guidelines for the spatial arrangement of displays in [[Simultaneous Interfaces]], how to choose between [[Embedded Interfaces]] and [[Separate Interfaces]], and evaluate interfaces at the interface-factor level rather than at the interface level, where there are too many confounds.
!!!!Comments & Questions
*Study Metrics
**This meta-analysis did not consider [[Multi-Level Interfaces]] which visualized 3D data, nor did they consider data representation formats as a factor in their survey.
**The studies surveyed did not consider the visual-spatial ability of study participants (their technical background or expertise is not reported in the meta-analysis). Objective study results were considered, albeit limited to time and accuracy results. Author insights were also used to support these results.
**Other objective and subjective measures were collected in some, but not all studies, including usage patterns, participant strategies (observed and self-reported), and interface choice (observed and self-reported). Often these results reinforce or explain the other objective measures reported by the studies. Eye-tracking tools were used in several studies, so as not to rely solely on self-reports. The authors acknowledge that visual attention and visual search interact in complex ways.
*Visual Levels
**Each visual level should add task-relevant information. What are the common characteristics of tasks that require different visual levels? With hierarchical data, the need is clearer.
**The benefits of high-level regions in [[Embedded Interfaces]] are often disputed. Section 6.5 discusses the limited roles of [[High-Level Displays]], despite claims made in the literature, but this is not specific to high-level regions in [[Embedded Interfaces]]. In Section 8.1, benefits of [[Embedded Interfaces]] are threatened by the issue of distortion surrounding the low-level region.
***[[High-Level Displays]] provide navigation shortcuts and a sense of data structure, but claims that they aid orientation in the data and provide meaning for the data have yet to be supported. What about gauging data reliability/integrity - can this be established from [[High-Level Displays]]?
**No studies report data density in [[High-Level Displays]]. What is design consideration for density: physical item density vs. task-specific density. How do you quantify ''too much information''?
** w.r.t. the readability of text in [[High-Level Displays]], what are the subjective responses? Is there a disconnect between objective and subjective results?
*Study Design
**In ch. 6, how is ''distraction'' accounted for, qualitatively defined, and quantified?
**Section 6.3 emphasizes the difficulty of designing tasks specifically for [[High-Level Displays]].
**Tasks in all of these studies have clear objective measures: time and error rate: data retrieval, comparison, path connecting and verification, co-occurrence searches. Trend and outlier identification in one paper only (Saraiya et. al. 2005), which are difficult to realistically assess with time and error rate alone - these tasks require acting on suspicions and drilling down for further inquiry and verification.
**How do you quantify ''distortion''? Subjectively and objectively - do these measures often match? How do you quantify ''confusion'' and ''disorientation''? Is ''constrained'' or ''predictable distortion'' a categorical or quantitative value?
*Findings
**Graphical (non-text) representation in [[High-Level Displays]] remain undefined.
**[[Temporal Interfaces]] place a strain on memory and result in a loss of context? How do you quantify or measure one's sense of context? How far must context be defined to be useful? Shi et. al. 2005 relies on the limits of memory for their findings; they report time, but not error rate.
**The choice between [[Embedded Interfaces]] and [[Separate Interfaces]] remains unclear based on study results. The data structure often biases the choice of interface and often confounds with the factor of the choice of interface.
**Authors claim that view coordination is difficult in [[Separate Interfaces]], but do not discuss why it is difficult, or different view coordination strategies (such as [[Linking]] or [[Brushing]], which are mentioned once in the concluding section). How do you quantify the amount of coordination between separate displays?
**A design trade-off: [[Temporal Interfaces]] afford familiar interactions (i.e. [[Panning]], [[Zooming]]), but have high-memory load. [[Simultaneous Interfaces]] often require complex interactions and view coordination (but have low-memory load). Memory load is a task-dependent factor in itself, as is familiarity, and both interact with expertise and technical background.
*Limitations
**No meta-level statistical analysis was conducted across all studies as tasks, data, and experimental designs varied considerably. Instead individual results from each study were tabulated in support of qualitative claims regarded interface design choices.
**The authors did not repeat the tasks using the systems described in the original research papers; they relied on descriptions of tasks and results, which they may have misinterpreted.
**Design and evaluation choices are often incomplete or absent in original research papers. Inadequate reporting in original papers. The authors could not relate results to existing theoretical frameworks (i.e. [[Information Foraging Theory]]).
!References
<<bibliography>>
!!!Conceptual Understanding
''Multidimensional Scaling'' (MDS) is good for finding structure in multivariate data. This involves scaling the high-dimensional data to a lower-dimensional space using a linear or non-linear combination of the original variables. Once this is done, the data can be plotted and if structure exists in the lower-dimensional space, it may be apparent. Beginning with a dissimilarity matrix, the dimensionality reduction is a projection of the data to a lower-dimensional space, preserving pairwise distances as well as possible, measured by some objective ''stress function''. There is often a balance chosen between a small number of dimensions and a desired level of stress.
However, there can be problems with the plots: there can be multiple ''local minima'', the input data may be ''indifferentiated'', the scaling algorithm and input may leave ''artifacts'' in the plot (false positive structure), and it may be difficult to make judgments regarding local structure (whereas you can make judgements regarding global structure). There may however be local dimensions in meaningful subsets.
With regards to the development of MDS visualization tools, providing interaction to intervene in the optimization process as the algorithm finds low stress values is recommended while visualizing interim states. This animation allows the observer to visually recognize convergence as it occurs. Allowing the manual editing of groups of points and rerunning the optimizer can also help to navigate local minima.
Metric MDS is more prone to ''local minima'', but less prone to ''degeneracies'', than non-linear MDS. In ''Metric MDS'', dissimilarities are continuous, while in ''Non-metric MDS'', dissimilarities are categorical or ordinal.
''Indifferentiation'' occurs when every object is equally dissimilar to every other objects, or the dissimilarity graph clusters around a single positive constant value. A form of ''null data''. This output (in 2D, two concentric discs of increasing radial density from center to perimeter, in 3D, a hollow sphere) is indicative of no structure in the input. Uniformly random dissimilarities (noise) are also the cause of indifferentiation. Noise is averaged out in nonlinear MDS, and thus the resulting plots appear similar to ''constant dissimilarity''.
!!!Implementations of MDS
*''~FastMap'' - fast MDS (linear complexity), operating incrementally: pairwise distances are computed using cosine similarities. Objects are projected onto a (//n- 1//) dimensional subspace
*''ISOMAP'' - when points have a complicated, non-linear relationship to one another, MDS and PCS break down. ISOMAP can handle such data. A 2D data set in a 3D space (i.e. the [[Swiss Roll]] surface), nearest-neighbours are computed using the geodesic distance (along the surface), rather than using Euclidean distance
!!!Terminology I don't understand
*monotone decreasing transformation / monotone transformation
*polynomial / monotone regression
*diagonal dominance
*eigenanalysis
*non-metric ~Kruskal-Shepard scaling
*grand tours / manual tours
*isotonic transformation / isotonic regression
*Minkowski metric / Lebesgue distance
*Procrustes method
*eigendecomposition
*strongly nonlinear transformation
*regular simplex
*minimum configurations
*nonlinear principal component analysis
*dominant eigensolutions
*degenerate transformation
*stability-favouring theme
*decoupling
*horseshoe effect
*null structure of indiscrimination
*correspondence analysis (chi-squared analysis) for multivariate categorical data
!!!Sources:
<<cite Buja2002 bibliography:Bibliography>>
<<cite Holbrey2006>>
<<cite Munzner2011>>
<<cite Tan2006>>
Like "Big Data" and "Data Science", "Visual Analytics" is one of those hard-to-define terms. At least in academic circles, it generally refers to the confluence of information visualization, interaction design, machine learning / applied stats, and applied cognitive science. Unfortunately, it often tends to get a lot of criticism from those domains: that VA research is out of touch with modern advances in ~InfoVis and ML, and it's too applied for core ~CogSci / perceptual psych people to care about. I often find myself split between ~InfoVis and VA.
Regarding machines "visualizing like humans": machines can understand and process raw data, they can classify it and make predictions better than humans can in many contexts. Humans are better at dealing with information, abstracted representations of data, which is why we visualize data as abstract shapes and patterns on screens. We transform data into abstract representations because humans have the power of analogical reasoning and the ability to think abstractly, which machines still have a hard time doing. This is why all VA is based around "humans in the loop".
On the other hand, there's been some success in teaching machines to perceive data like humans do. Back in the 70s and 80s statistician John Tukey coined the term "scagnostics" or scatter plot diagnostics: describing what makes a scatterplot of data "visually interesting" to humans. Is the pattern clumpy? stringy? correlated? are their outliers? More recently, Leland Wilkinson at U. Illinois Chicago, has been applying the idea of scagnostics in VA: give the system a lot of data, and it will search for visual projections of data that humans might find interesting. It turns out the idea generalizes beyond scatterplots to other representations of data too, such as time-series curves.
And then there's computer vision. From what I know, biological vision and computer vision are mechanically different, similar to how traditional artificial neural networks turned out to be quite different from biological neural networks. Here's another instance where machines are rapidly improving in their ability to process and classify raw data: air-traffic control, security, self-driving cars, radiology, assembly line systems are all benefiting from this. However, there's still a huge gap between showing a machine raw data and showing a machine an abstract representation, which is where humans still excel. For instance, can a machine understand and appreciate an abstract expressionist painting? Probably not, but it would be able to classify an photograph as having a cat in it.
A data-driven thematic coding or tagging technique for identifying themes in rich qualitative data. Highly sensitive to the data collected, the themes and code set are derived directly from the data. Resulting coded data can be subsequently analyzed statistically to isolate the most important themes. Also see [[Grounded Theory]].
!!Literature Notes & Commentary
<<cite Mathison1988 bibliography:Bibliography-EPSE595>> on Triangulation
*types of triangulation: space, time, and person, investigator, theory, and methodological
*results: convergence, inconsistency, contradiction: task of researcher is to take each as evidence and construct plausible explanations - triangulation is a means, not an end;
**reliability not to be confused with validity;
**just as individuals can be subjective, individual methods are also subjectvive
*Triangulation of methods in ~C-TOC interruption study: experiment, questionnaire, semi-structured interview - inconsistencies with self-reports in interviews and experimental results, some convergence between questionnaires and experimental results
**triangulation of experimenters: MB + CL
*Triangulation (methods, investigators) in haptic crayola project: experiment, questionnaire, 4 experimenters
*Triangulation in analysis in the [[DRITW|HDD-DR Ethnographic Project]] project: MS + MB, inter-rater reliability; convergence and inconstancy largely dependent on familiarity with subject area - novice / advanced coder
*mixed methods evaluation as prescribed by [[MILCs]], evaluation in large company contexts (Sedlmair 2011), and by Lam et al.'s Seven Guiding Scenarios (Lam 2011)
<<cite Ryan2003>>
*''scrutiny techniques'': themes arise from repetition, indigenous typologies or categories, metaphors and analogies, transitions, similarities and differences, linguistic connectors, missing data, theory-related material
*''processing techniques'': cutting and sorting, word lists and key word in context (KWIC), word co-occurrence, metacoding,
*selecting techniques: what kind of data is it? expertise of coders (mixed is good)? how much time / labour can you afford? number and kinds of themes (breadth/depth)? reliability and validity (inter-rater reliability/validity? ask respondents to comment on themes)
*Use of MDS for qualitative data analysis - could be somewhat circular to apply to qualitative data about the use of MDS in [[DRITW|HDD-DR Ethnographic Project]]
**some in-vivo coding (i.e. DH's in-house DR techniques, bioinformatics jargon)
**theory-related material (influence of initial Swiss roll manifold hypothesis)
**Many of the techniques (and rationale for using them) are contingent upon having large amounts of raw textual data: interview transcripts being superior to field notes or interview notes, as the latter is already a preliminary (and theoretically sensitive) thematic coding.
**we did not have transcripts but field notes - limiting us to searching for repetitions, similarities and differences, cutting and sorting - however indigenous typologies were also noted
*refs to read:
**Barkin, S., G. Ryan, and L. Gelberg. 1999. What clinicians can do to further youth violence primary prevention: A qualitative study. Injury Prevention 5:53ꪪBerelson, B. 1952. Content analysis in communication research. Glencoe, IL: Free Press.
**Charmaz, K. 2000. Grounded theory: Objectivist and constructivist methods. In Handbook of qualitative research,2nd ed., edited by N. Denzin and Y.Lincoln,509䨯usand Oaks, CA: Sage.
**Jang, H.-Y., and G. Barnett. 1994. Cultural differences in organizational communication: A semantic network analysis. Bulletin de M䯬ogie Sociologique 44 (September): 31ꪪShapiro, G., and J. Markoff. 1997. A matter of definition. In Text analysis for the social sciences: Methods for drawing statistical inferences from texts and transcripts, edited by C. Roberts, 9͡hwah, NJ: Lawrence Erlbaum.
!!References
<<bibliography>>
<<list filter [tag[overview]]>>
interview date: 12-12-04
story: [[Ryan asked for federal help as he championed cuts|http://bigstory.ap.org/article/ryan-asked-federal-help-he-championed-cuts]]
behind-the-scenes: [[Document mining shows Paul Ryan relying on the the programs he criticizes|http://overview.ap.org/blog/2012/11/document-mining-shows-paul-ryan-relying-on-the-the-programs-he-criticizes/]]
!!On Overview
!!!Utility & efficacy
> @@color:#0000bb; What [have / are] you [used / using] Overview to do? @@
Can Overview ingest some or all of your data? How long did this take? How much time did you spend cleaning/formatting the data specifically for Overview?
''JG'' received the data in chunks, as FOIA requests were answered, delivered via ~FedEx on paper (90%) CD (<10%); others were found on an FTP server (<10%). More is still being delivered. At the time of the Ryan story, there was about 9,000 pages of material from a handful of government agencies. Not all documents were email/correspondence; some were fax headers, transcripts, press releases, meeting minutes/attendees, insurance policy conversations, reports/briefs, FOIA request cover sheets. Many had significant OCR errors.
A few days of scanning documents, followed by uploading, then bifurcating the documents into individual pages and formatting into an Overview-compliant CSV file via a Perl script. Most of the time spent on this project was in this phase.
''AP'': For each file, he scanned the pages electronically and uploaded them to the APrnal 䯣s�ent Cloud server,
''JG'': ﬠnot sure if this is useful, but attached is a Perl script I wrote (I was too lazy to do it in a modern language) that parses documents uploaded Document Cloud, grabs their OCR results and splits them into separate 1-page chunks and formats them into the CSV file that Overview uses
''AP'': By the time Gillum sat down to write, he had received more than 8,900 pages (and counting, since the government continues to send additional documents every day).
> @@color:#0000bb; Did you have hunches about a story a priori? Were you able to verify these hunches? @@
''AP'' : Gillum already reported separately in mid-August that Ryan, despite his denials, had lobbied for millions of dollars in economic stimulus money, even writing Vice President Joe Biden for help.
[Overview] helped Gillum identify instances in which Ryan had sought government funding in the form of expanding food stamps, federally guaranteed business loans, grants to invest in green technology and money under President Barack Obama塬th care law _ the kinds of government largess that Ryan is now campaigning against.
The standard process is to vet new political candidates or appointees, to determine the types of things Ryan asked for in correspondence with various government agencies. 䠩s he asking for? How is he behind the scenes?�ng evidence of hypocrisy was not a hunch in itself during the use of Overview.
> @@color:#0000bb; Did you develop new hunches about your story while using Overview? @@
Once the types of requests that Ryan made were identified - these were in a small subset of the full dataset, one of the 8-node level branches under the orrespondence� individual documents were annotated within Document Cloud. These notes served as flags / follow-up pointers for further investigate reporting. At this point, the annotated correspondence would be compared with campaign statements, or statements on these issues would be requested from Ryan᭰aign staff for his position on a particular issue. It was at this point that hypocrisy was identified and collected.
Overview helped JG to separate relevant from irrelevant, as well as find clusters that were the result of OCR errors that would have escaped keyword searches in Document Cloud. There werenclusters that were examples of a keyword that he wouldnᶥ searched for in Document Cloud.
Overview also helped him to identify clusters of documents that were heavily redacted, often by particular agencies. This is driving continuing investigations into these agencies.
> @@color:#0000bb; How did you explore/search your data using Overview? How long did this take? @@
50% of the time was spent exploring the entire 9,000 pages at once, and the remaining 50% of the time was spent examining 4 smaller chunks/subsets of the full dataset, the availability of which was subject to when ~FOIA-requested document sets physically arrived at the Washington AP bureau. The smallest of these chunks was ~1,150 pages, the largest around 6,000. JG realizes that this may have been an inefficient way to go about analyzing the full dataset, but due to the time pressure involved to get the story out before the election, he could not afford to wait for the full document set.
Exploring the full document set and broadly tagging took the better part of an afternoon.
> @@color:#0000bb; How did you choose which documents to read? Depth vs. breadth? How long did you spend employing these strategies? How many individual documents did you skim / read? @@
At the size-8 granularity of the tree view, a leaf or level-2 node would be selected, so at least 8 documents were listed. He proceeded to page down and skim the documents in the node (which were at max a page in length). Some branches could be easily dismissed this way, after a few documents, the whole branch could be tagged. This was repeated in a left-to-right sweep of the tree. Otherwise, he would lasso the documents of the currently selected branch in the Items plot, along with their neighbouring points, which would indicate where those neighbouring points fell in the Tree layout: this was 멮g for stray documents詳 was what motivated drilling down in the orrespondence�hes. Occasionally, the Select Random feature was used to verify if the tag correctly characterized the selected branch.
> @@color:#0000bb; How did you go about tagging your data? How long did this take? How many tags did you generate? How many did you delete? How many did you consolidate / split? How many of these tags were structural vs. semantic? @@
The higher-level tag set from the full 9,000-page dataset was reflected in the smaller subset datasets. Some detail was added to these tags for the smaller subsets, but generally there were about a dozen higher-level tags, recreated for each subset; tag names and tagged documents were not propagated from the full dataset to the subsets (it is possible to load a tag file created for another dataset, and if there is overlap between document identifiers in the current documents set and the tag file, these will become tagged). It appears as though tags were recreated often as familiarity with the structure of clusters in the document set became clear.
''JG'': It was a proportional tags-to-documents ratio, I believe. The tags themselves didn't really change much -- that is, the topics/"bins" I wanted to keep things in -- at least to my knowledge.
> @@color:#0000bb; What level(s) of tree pruning were used? 16? 8? @@
Most of the time was spent at the size-8 node in the Tree View. An exception was the orrespondence�h, which was of greatest interest. Here, JG drills down to size-4, size-2, in pursuit of branches not fully described by the tag given at the size-8 granularity. These lower cluster-size nodes often had multiple tags (the parent branch᧠along with a more specific tag).
> @@color:#0000bb; How long did you use Overview for? How long was spent reading documents vs. organizing, browsing / sorting / tagging? @@
>> @@color:#0000bb; Did this match your timeframe expectation prior to using Overview, given the size of the dataset? @@
>> @@color:#0000bb; How much time was spent reading a single document? Did this vary? @@
Organizing, sorting, and tagging the full document set took place over the course of an afternoon. Annotating the documents in DC took about 2 days. Most of the subsequent time (~2 weeks) was spent following-up on annotated documents, seeking out campaign statements or stated positions, comparing and contrasting.
Documents were of fixed size (max 1 page), and given an 9,000 pages explored broadly over the course of an afternoon, it appears that only a small subset of documents outside of the orrespondence�h were skimmed.
> @@color:#0000bb; What proportion of documents did you read/skim? Are you confident in this proportion? Why or why not? @@
A potential follow-up question to clarify. Most of the read documents were in the orrespondence�h (~200 pages). Others were skimmed. The intent here was to Სte the wheat from the chaff� @@color:#0000bb; What proportion of documents did you tag? Are you confident in this proportion? Why or why not? @@
It seems that most of the orrespondence�h was tagged, but this should be verified; the final tag file only accounts for ~8.2% of the entire dataset of 9k pages.
> @@color:#0000bb; Not enough tags vs. too many tags? @@
> @@color:#0000bb; How did you deal with [unique / weird / important / unimportant / relevant / irrelevant] documents? (may be overlapping categories) @@
>> @@color:#0000bb; How did you dismiss the unimportant? The irrelevant AND unimportant? @@
''AP'' : Gillum cared little about 300 pages on a constituent's problems with a credit card company, but he was interested in letters asking for agriculture subsidies
There was a higher-level branch of fax cover sheets, insurance policy requests, neither of which were informative. Some of the documents scanned and ingested pertained to another potential political candidate, so were irrelevant to the current story; nevertheless these each received higher-level tags.
> @@color:#0000bb; Did you ever flag documents for later follow-up? Did this eventually happen? @@
''JG'' flagged documents by annotating them in Document Cloud, which could be done either in the browser or in the embedded web viewer. These were starting points for further investigative analysis, to compare against campaign statements that illustrate a position on a particular issue.
> @@color:#0000bb; Did you ever re-read documents? Was this intentional? @@
> @@color:#0000bb; How, if at all, did you use the sampling functionality to read documents? @@
Yes, to verify the tag correctness / consistency from the root of the branch downwards, to 㨠anything that might have been missed�thing deserving of a more specific tag.
> @@color:#0000bb; Context of use: where were you while using Overview? What other applications did you have running? Were you taking handwritten or digital notes? Were you using Overview alone or in collaboration with others? @@
Overview was used by JG on several machines (Mac and Windows) at the AP Washington Bureau. Notes were recorded in Document Cloud. JG was the primary Overview user; him and his team would follow-up on the notes he created in Document Cloud.
''JG'' : I had been working on a few different machines and breaking up the documents into chunks with no apparent organization (and as the FOIAs came rolling in), which I guess was fairly sloppy on my part.
The end result was to tag notes in ~DocCloud that I could later pull from for reporting, so I guess I was satisfied in achieving that goal.
I also made the mistake of deleting - twice, mind you - a folder with all of overview's files, so I had to keep regenerating the files.
!!Usability
> @@color:#0000bb; Let's discuss Overview's usability issues. @@
> @@color:#0000bb; On installing Overview @@
Thereשndows Java VM incompatibility issue, which makes Overview difficult to run/install on some Windows versions.
> @@color:#0000bb; On preprocessing data / ingesting data @@
JS acknowledges that the data preprocessing and ingestion pipeline is where most of the known usability issues/barriers are.
Would have wanted to detect cover pages, as in JS⡱ contractors dataset, rather than splitting the documents at the page level. However, the OCR was really terrible in many cases.
> @@color:#0000bb; Your first use of Overview: Orienting yourself within the UI @@
This took about a day.
> @@color:#0000bb; [Overview-specific question] Usability issues w.r.t: @@
>> @@color:#0000bb; Linked displays, selecting items @@
>> @@color:#0000bb; the tree display @@
Requested an ability to hide/expand cluster nodes, and the ability to pan and zoom. (These are afforded in the new Web version; privacy concerns ruled out use of the web version until now, since JGments had to live on the APrnal DC instance, not on the public-facing DC. The current update (Nov pports local ~CSVs and URLS external to the public-facing DC, the ability to run Overview in a virtual server. This makes the web version more attractive to JG.)
He wanted a way to ignore and re-cluster the document set for only a selected branch, but he realizes the mathematical implications of this, that measures of similarity would have to be adjusted and may no longer be as meaningful than when considering the entire document set. Hiding and showing nodes in the current web version may suffice.
> @@color:#0000bb; the MDS display @@
> @@color:#0000bb; UI for sampling @@
> @@color:#0000bb; UI for tagging @@
> @@color:#0000bb; UI for reading / opening in the browser @@
There is a issue with DC in that JG wanted a way to view all notes for the whole dataset together (currently the document set is split across several chunks, meaning several DC document sets, and thus notes can only be collected and viewed for the subsets, not altogether for all ~9k pages.
> @@color:#0000bb; What UI features did you like? Which didn't you like? @@
!!Learnability
> @@color:#0000bb; Related to usability, lets talk about learnability and Overview. @@
> @@color:#0000bb; Learning materials: self-exploration vs. relying on JS's blog posts and instructions, other sources? How much time spent on each? @@
''JG'' was familiar with JS⡱ war contractor dataset and workflow.
> @@color:#0000bb; Developing a conceptual understanding of item layouts in the MDS display and the tree display. @@
Given his technical background and his acknowledgment of the mathematical implications of re-clustering after filtering the majority of the dataset, itᦥ to assume that JG has a strong conceptual understanding of how items are laid out in Overview.
!!Comparison: before and after Overview
!!!Improvements
> @@color:#0000bb; Has Overview improved your process? Do you see Overview as having the potential to improve your process? @@
It definitely sped up the process, and time was critical given the election.
!!!Problems left unaddressed
> @@color:#0000bb; What can't Overview do for you? What problem remains unaddressed? @@
Aggregating notes in DC across multiple documents.
!!!Previously untouched data
> @@color:#0000bb; Were you able to ingest/analyze data that you couldn't approach with other tools? @@
All these documents were already in DC, however poor OCR made some of these documents non-machine readable, making it difficult to search for them in DC.
!!!Previously unapproachable tasks
> @@color:#0000bb; Were you able to perform tasks with Overview that you couldn't do (or couldn't do efficiently) with other tools? @@
Verification of OCR documents couldn堤one easily before; previously he relied on keyword searches in DC, browsing and skimming the list of results.
!!!Discoveries with Overview
> @@color:#0000bb; Have you made discoveries (in your data) using Overview that you wouldn't have been able to make with other tools? What type of knowledge does Overview generate? (somewhere on a continuum between a summary and locating specific documents). @@
''JG'' was able to find OCR errors that would have escaped keyword searches in DC. New categories of Ryanﲲespondence topics were not discovered. These documents can still be categorized by a human.
!!Adoption
> @@color:#0000bb; Has Overview become part of your workflow? @@
Yes. Haven㥤 in the past 3 weeks, but planning on it for future similar stories. May switch to web version.
> @@color:#0000bb; [if not] do you foreseeing it becoming part of your workflow? Will it replace / add to / complement your workflow? @@
Currently planning to use Overview for investigating redacted information from various agencies.
> @@color:#0000bb; How often is there a document set where Overview could be used? @@
Whenever a political candidate / appointee becomes subject to FOIA requests.
> @@color:#0000bb; Would you recommend use of Overview for colleagues? @@
>> @@color:#0000bb; Do you expect them to you use it in their workflows? @@
There are still barriers to use in terms of installation and configuration (particularly on Windows with its Java incompatibility), data cleansing and formatting, and ingestion into Overview.
!On data journalism and document mining
!!Who: background & expertise
> @@color:#0000bb; What brought you to working in this type of journalism? @@
Wanted to make a difference.
!!!Domain expertise
> @@color:#0000bb; Relevant demographic information: age, number of years working in journalism, number of years working in data journalism, education background @@
''AP'' : Gillum, the data journalism expert on Washingtonstigative reporting team
!!!Technical background
>> @@color:#0000bb; Re: data journalism: self-educated vs. formally educated, trained, or mentored, mixed (discerning between what was formally taught and what was independently learned) @@
>> @@color:#0000bb; Technical skill set: spreadsheets and table manipulation, data cleansing, internal and external validation; programming / scripting experience @@
>>> @@color:#0000bb; skill set w.r.t. document mining and unstructured data vs. skill set w.r.t. structured data @@
CS/Physics undergrad, recent Masters in Journalism (JS could have been an instructor). Attends NICAR, exchanges tools and datasets with others in that community.
!!! Context of data journalism
> @@color:#0000bb; Tell me about your current day-to-day work. @@
!!!! Context
> @@color:#0000bb; Career context: agency / bureau affiliation, past and present @@
> @@color:#0000bb; Spatial context: where does the work happen? (office setting, working remotely, other) @@
The AP᳨ington bureau, with a team of investigative reporters.
> @@color:#0000bb; Task context: what else is going on? multi-tasking vs. single-task focus? @@
> @@color:#0000bb; Temporal context: project timeframe + deadlines: hours, days, weeks, months, vs. ongoing investigation @@
Stories like this are investigated over a 2 week period in which they are racing against other agencies, such as the NYT.
!!!! Collaboration
> @@color:#0000bb; the extent to which work is collaborative @@
> @@color:#0000bb; w.r.t. data acquisition, data pre-processing, analysis, writing @@
> @@color:#0000bb; reliance on technical expertise of colleagues @@
!!! Workflows and processes
> @@color:#0000bb; I'd like to hear about a recent story of yours, one from before using Overview, involving large collections of documents, representative of this type of work. @@
>> @@color:#0000bb; (If there is no precedent, can I hear about a story involving large amounts of data analysis?) @@
>> @@color:#0000bb; Is this example representative or unique? @@
>> @@color:#0000bb; (please provide links to articles, or send manuscripts) @@
''JG'': The SNA [Social Network Analysis] I used was sort of supplemental to FOI requests of perry's phone records, which were fax-quality documents that needed the use of Mechanical Turk to digitize. The SNA came into play when I took lobbying and campaign contribution data from texas and the Federal Election Commission for Perry, standardized names and put them into a subject-predicate format. It, at the minimum, helped me figure out who, at least visually, was fairly well-connected in terms of whom perry contacted by phone.
Here's the story: [[Perry Called Top Donors from Work Phones|http://www.boston.com/news/politics/articles/2011/12/06/perry_called_top_donors_from_work_phones/]]
!!!Methodology
> @@color:#0000bb; data collection and analysis methodology prior to Overview: personal vs. adopted / taught? strictly followed vs. ad hoc variations, subject to constraints of data, deadlines? @@
''JG'' can write code and scripts for data cleansing and bifurcation, uses ~ArcGIS, SPSS on a regular basis, familiar with VA tools, notably those for social network analysis (as used in a previous story on Gov. Rick Perry, use of ~MTurk).
!!!Reader perspective vs. peer perspective
> @@color:#0000bb; How should readers react to document mining in journalism? What do the reader comment boards say? @@
> @@color:#0000bb; How should document mining in data journalism, its goals and processes, be reviewed and critiqued be peers? How has your data journalism been reviewed / critiqued? @@
''AP'': Jack Gillum is a campaign finance and investigative reporter for The Associated Press. Gillum joined the AP in July 2010 from USA Today, where he reported and wrote data-driven investigative projects. Among his stories: an examination of testing irregularities in Washington, D.C. schools as the system trumpeted reforms and a look at how taxpayer subsidies play a growing role in university sports programs. Working from the AP's Washington bureau, Gillum has been covering the role of money in campaigns throughout the 2012 election cycle at a time when court rulings have radically changed how outside groups can influence elections. Before joining USA Today in 2008, Gillum worked as a business reporter and database specialist for the Arizona Daily Star in Tucson, Ariz. He is a graduate of Santa Clara University and has a master's in journalism from Columbia University.
story : [[TPD working through flawed mobile system|http://www.tulsaworld.com/article.aspx/TPD_working_through_flawed_mobile_system/20120603_11_a1_cutlin136616]]
behind-the-scenes: [[How the Tulsa World Used AP϶erview to Report on 8,000 Police Department Emails|http://www.pbs.org/idealab/2012/06/how-the-tulsa-world-used-aps-overview-to-report-on-8000-police-department-emails177]]
[copied from initial recording - 12.06.07]
[00:00:30] JW >> You want to check out the news room?
[00:00:31] MB >> Yeah, this is part of it, so.
[00:00:34] JW >> Kinda have a very open news room, this is the whole place. We have all our sections all on one floor.
[00:00:42] MB >> How many people work in your newsroom?
[00:00:46] JW >> Uh newsroom, I think somewhere, I'd say 80 - 100.
[00:00:52] MB >> OK.
[00:00:53] JW >> So we're a, let's see, my numbers may be a few years old, but I know we used to do 200,000 or so a day and 240 [thousand] on Sundays, something like that. Pretty solid, medium-sized paper.
[MB explains my role in the research, my use of Overview]
[starts video recording, time reset]
[JS joins conversation, adjusts microphone]
[off camera, MB asks permission to record, asks JW to prepare to show us his exploration and analysis process with Overview]
[00:00:14] JW >> Alright, so should I?
[00:00:18] MB >> Yep. Perfect.
[00:00:18] JW >> Alright cool.
[00:00:24] [screen sharing starts]
[00:00:29] JW >> I left or went ahead and didn't load the tags so that I could kinda show you my first take at it, from the start. Also another thing that Jonathan and I realized later was that working on a square screen I suddently had a criticism for him early saling like "it's kinda pushed together". It wasn't until I saw a screen shot of his screen on a wide screen and I'm like "oh that's how it should be". But this is on a square screen.
[00:01:03] Alright so basically when I first loaded it up this is what I had. Immediately I had to start somewhere, 㴡rted clicking around. Immediately I was looking for things I could dismiss, that was kinda my first issue was is this kinda working toward emails and subject matters I could just dismiss.
[00:01:38] MB >> [screen paging issue w/ Jonathan]
[00:02:00] JW >> Alright walell I don't really remember what were the first couple of things I looked through, but I do remember being able to immediately look at groups of documents and just find things that I wasn't interested in. There's a huge section of堤o a lot of calendar scheduling with their emails, so I was able to able to get rid of a lot of emails just through confirmations for meetings and things like that. By get rid of I of course mean tag 02:41] MB >> So how did you find those? Did you navigate through the tree to find those?
[00:02:49] JW >> I mostly used the tree. The only thing I ended up using the graphics plot for was later on I had unread emails and it was easier to find the last oh, hundred unread emails through the clusters and just pick out the grey dots 배:03:15] So this branch right here is great because this is all service desk requests. These ended up being every time somebody had a problem they appear here, they had to fill out a service desk request so because this project is about problems with computers there's lots of these, and of course it's just very nice to go ahead and tag them and have 'em for later if I need 'em.
[00:03:46] So basically I mean the first day or two was just trying to figure out a plan of how to attack this 䨩nk once I had explored it I got a sense of what it [the tree nodes] it was these down into. That's when I started trying to look at nodes as broadly as possible and just try to go ahead and tag them, stick them in a place where I could read them individually later. So that worked out pretty well.
[00:04:29] My one mistake �oing to load my tags now �e definite mistake was that I had way too many detailed tags to begin with. [Navigates to tag file] If you guys have loaded the tags I'm sure you agree there's just a ridiculous amount of tags. So later on I as I went I got more toward where I wanted to go with them. Even at the very late stages ended up putting a "Trash" tag on which I kinda regretted but it was an easy way to uh to kinda get the errant emails the ones that just kinda were rogue and didn't belong to any category. It was an easy way to at least let myself know that I had already looked at it, and I didn't have to look at it again.
[00:05:38] MB >> Did that include all those "Service Desk" documents? Or did you tag those differently?
[00:05:44] JW >> the "Service Desk" ones mostly identified early. But then again service desk ones kept on popping up like as I said in a rogue fashion, so later on a lot of those probably ended up in the "Trash". Basically at the end I was trying to make sure the unread emails were at least addressed, wheareas in the beginning I could just throw a group of unimportant emails into a tag and later on I just wanted to make sure I wasn't re-reading unread emails. So they were either going into the trash or one of my other significant categories I guess. I never really did think of a great system for tagging, but then again it was always "a tag is better than no tag". Complicated or disorganized tags didn't matter as long as they [the emails] were tagged. It's kinda what I realized later.
[00:06:50] MB >> So do you think the number of tags you created was a product of how you explored the tree view?
[00:06:59] JW >> Definitely, as I started out at 32 [node size] I went down to 16 䠡nother thing I found out as I went down in node iterations I started going left to right and I'm not sure about the way it was created, why left to right, it seemed to have an organization factor to it, but what I could do, every time I went down I could go left to right and find things that I hadn't read before and go ahead and retag them and it seems like more work than was necessary but at the same time it felt very organized, a very organized way of get where I needed to go, it was to keep on going left to right. A lot of the time, there wouldn't be any emails over on the left side it was just more and more emails showing up on the right side so it seemed smart going left to right. I'd say the stage that I spent the most time on was getting down to 2 [node size] or so and trying to explore all of - and maybe Jonathan you can explain that; was there a reason that over on the - when I went down that over on the right there was lots more unique emails that didn't fall into any other trees?
[00:08:27] JS >> You mean why they would be on the right as opposed to the left?
[00:08:31] JW >> Yeah it seemed like on the left I didn't run into any groups of emails that didn't make it into the 8, 16 [node size] iterations but it seemed like there was more and more over on the right.
[00:08:47] JS >> I didn't write the tree layout algorithm, that's a Stephen [Ingram] concern question. This is very interesting though. It's fascinating watching other people use software you worked on. This is not how I was imagining it would be used. Keep going and I'll tell you how I thought I'd have done it at the end.
[00:09:14] MB >> And it's a question I can ask easily to Stephen [Ingram] later today, I'll be seeing him in a meeting.
[00:09:20] JW >> I mean I definetely kind of got some early help and some early tips from Jonathan, and when it came down to exploring this and tagging I just at the worst I wanted to use this as a way of organizing me looking through every email and at best I wanted to look at most of the emails. I guess, does that make sense?
[00:09:46] MB >> [affirmation] So it was a very systematic sweeping approach in that sense.
[00:09:50] JW >> Like 确ted to make sure that no emails fell between the cracks and this program seemed to suit both if for instance I knew I wanted to read every email I would still use Overview because it organizes in a way that you could read every email quick. If I 䨩nk maybe what Jonathan's getting at is the way that it was intended was that you should be able to get to the emails that you want to read and I think it serves those purposes. I just couldn't � Jonathan could help me right now thinking about it but I couldn't find a way to bridge those gaps, to read most of the emails and not dismiss or miss some that fell over into another category. Jonathan, what do you think?
[00:11:00] JS >> I'm trying to make sure I understand what you're saying here - you very much wanted to read every email to make sure you didn't miss anything, is that the substance of what you're getting at?
[00:11:19] JW >> I guess I meant to focus on reading and dismissing what I can and then getting every single email to make sure I had 岹 single rogue email, the ones that didn't fall into any other category. So to get to those I ended up having to do that left to right thing I was talking about which was go through and make sure that I was getting every single node, checking out every single unread email. It was great, actually, I mean didn't really emphasize that is was great, for instance the service desk emails: gone, didn't have to read those after glancing at them, and probably another thousand or two fell into that same category, of being able to identify them and move on, but the breadth of my time, the most of my time was spent in the last couple of days of looking at Overview and just looking for unique emails.
[00:12:26] JS >> Yeah, that's actually really interesting. I think I'm hearing two things from that or at least that are relevant to 堤esign of the software going forward. One is that you definitely wanted to look at every single node in the tree to make sure you didn't miss anything and the other is that you were able to take, in some cases, entire branches and dismiss them and say "I'm not going to read these".
[00:13:06] JW >> Yeah, I mean I guess the relevant point to make is that Overview definitely suited my purposes of the extreme of "I wanted to read every email" and the basic principle of being able to obviously not read every email, the whole purpose that I could glance at and dismiss without reading the 13:35] JS >> That's very interesting, of course we have the log so I think Matthew will be able to tell you at some point in the future how many emails you actually read.
[00:13:46] MB >> Well at least I'll be able to tell how many were opened. What I won't know based on the log is how many you glanced at and just skimmed a few key words, maybe browsing through it, but I can't really tell based on the log. I can infer based on the time between two successive documents being opened but I can't really tell if you were able to skim a document, say in those clusters on the right, or sorry the left of the tree, those bigger clusters like the service desk ones, versus the ones that branched later, it will be hard to tell. Maybe you can answer tha. DId you just glance at those emails from the big clusters on the left of the tree and did you tend to read more and more of the emails as you progressed toward the right?
[00:14:34] JW >> You know it's intersting because when I got to a branch that I deemed had interesting emails I just literally started going down just like this, just use your arrow keys and just glance, glance, glance [clicks through 3 emails in document list].
[00:14:59] MB >> "This is another service desk email, proceed to the next one", and so forth.
[00:15:05] JW >> Now the service desk emails, no I did not do that. I just used the keywords that are listed in the bottom left [document list] and every single one was the same, service desk type, then sign-on.
[00:15:21] MB >> Did you rely on the folder list at all for that? Above it?
[00:15:25] JW >> Let's see, I'm trying to think of what did I most use the folder list forᴠI most used the folder list for was when I got down to the lesser nodes, I could look at the folder list and see if there was one that didn't have any 䧳 see. [clicks on parent node, all nodes selected, corrects, selects cluster on left side of tree view]. Alright what I used the folder list later on was for looking for these untagged folders to find the unread ones cause when you get down there you have a nice little two-for node or folder and I could quickly tag those and move on. That's the most I used the folder list for.
[00:16:31] MB >> To what extent did you use the sampling, the 'Select Random' selection button?
[00:16:36] JW >> I didn't but I did later on when I was kinda doing a review of a tag, say [the] 'Important' [tag], I could just, do that, honestly to show a boss. I could say "these are my important emails, these are my relevant emails to the story that I think I have" and I could just say "all of these are good stuff", just 'Select Random', so I used it in a novel way.
[00:17:17] MB >> For communicating to your colleagues or to your editor or boss.
[00:17:23] JW >> You know it can't be understated that when my bosses wanted to understand what I was working on I could show them Overview and say "here's 3,000 emails graphically displayed in a way that you can understand and as a reporter that's asking for to 堳tory, showing them this was really great, it's not just a folder of PDFs but it's actually shows graphically exactly what I want.
[00:17:58] MB >> Out of curiosity, is the 'Important' tag the same documents that are linked to in the article, more or less?
[00:18:07] JW >> Many less. What I actually ended up doing to 篯d chunk of my 'Important' tags have notes attached to them, especially later on, so what I did over on the Document Cloud side since I realized that it's hard to get to, hard to flag documents on Document Cloud unless they're noted. Obviously putting an 'Important' or 'Relevant' tag in Overview doesn't help you on the Document Cloud side. So I started making sure that all of these that I ended up using had a note on them and then I could drag all of my annotated nodes from Document Cloud into a new project, and that project scuplted down a bit because my editors, my head editor, editor-in-chief actually didn't want to give too much of our records request away. So I understand what he was thinking about but I definitely agree 19:26] MB >> So at what point in this process between when you were sweeping from left to right, at which point did the story start coming out of this? I mean you had a hunch going in but where did the story start?
[00:19:42] JW >> The records request from a specific tip, the more things that I researched on that tip that we got about 6 months ago the more factual was. The one thing that had eluded us was the tip that these officers who had paid all this money for the system were going around, traveling around the states, going to conferences, getting into hotels, going out for drinks, getting all of that paid for. Going into this we knew we wanted to learn more about the police, get more insight into what was happening on the side of the city police that were purchasing this, but also we were hoping to actually get confirmation of travel and things like that. And all of the above happened. So as I'm going through this I kinda know what story I want and will probably get out of this, I think around here we like to discuss the minimum and maximum, what we'll at least be able to do with these emails and what will we most be able to do with them. I definitely got near the maximum, I got all of the above: travel records, way more insight than I thought that I was buying into the actual purchasing process and other things. So yes I got to where I thought we were going to. Jonathan actually said something earlier that I kept in mind the whole time which was "how do you know what you're going to write until you've looked at all the emails", which I know is kinda like the motto. That helped me keep an open mind as I actually read them, so it was definietly an interesting mix of I knew what I was probably going to get and actually did get, but then also I wanted to make sure that I get through all these rogue emails. and everything else before I settled on writing.
[00:22:07] MB >> and it definitely seems reflected that it's in your tips, number #7. It really shone out in your notes to me yesterday. That you can't get time to read them all in the first place but you were able to find what you were looking for which was really nice.
[00:22:28] So how much time had passed, I mean there's a gap in your chronology of your notes, just because you got busy obviously, but how much time had passed between your initial explorations and when you finally got to that story?
[00:22:46] JW >> Let's see, I think the timeline is that in April, we were supposed to get all of our emails April 1st, and actually strung em out to me gigabyte by gigabyte over about 2 weeks ੴ was the first or second week of April that actually we finally got all the records and then 驮g to get them uploaded to Document Cloud.
[00:23:17] MB >> Yeah I recall that email conversation.
[00:23:19] JW >> That was a really obnoxious thing because on my end there's too many different formats at play, so I think it was like mid-April or toward the end that I realized that the problem was totally on my end and to fix all of the above problems I just needed to cleant the Outlook emails. So we finally got them uploaded to Document Cloud, we finally had the CSV created, that all worked perfectly once I dumbed it all down, all the email text aside, and I think, whenever that was, April 19th-20th, 23rd, something like that, was when I actually started just reading the emails and like I said, it took a day or two of sort of exploring and figuring out the best way to start looking at them. It was just finding interesting nodes and reading all the emails, tagging them. At that point I was finally into the speed test that I wanted, I know that's important to you guys. Getting through these fast would justify the time I was spending on it to my bosses.
[00:24:45] MB >> Based on your notes it seems like you were doing a lot of this after hours in your own time.
[00:24:54] JW >> Yeah, when I got about a week or two away, my bosses said "Hey, we're happy with what you got, wipe off everything off your list, and just go to town on it". But when I was in the Overview stage it was definitely like they were supporting me but also I needed to keep a workload up and so they were telling me "take your time with this, just read them when you have time", but I was still obsessed with getting this done and looking for the emails, so I was definitely doing say about 6 hours other work a day, then reading emails for 4 to 6 hours. I was trying to get it to load up on my laptop too but that became more of a logistical problem than was necessary so I ended up just not being able to do it at home like was doing it here.
[00:25:59] MB >> So it was 4 to 6 hours for 2 weeks, then?
[00:26:03] JW >> Um, I think I got em all read in about 2 weeks.
[00:26:07] MB >> So that was, there was about 5,100 or something emails in there?
[00:26:16] JW >> 5,900-something.
[00:26:18] MB >> So your speed test, you got gradually faster and faster as you progressed toward the end? Or did you find yourself reading more toward the end, and actually slowing down?
[00:26:29] JW >> That's a good question. I don't know. I was going to say that looking through the emails, there was really no streamlining that I could do, it was all about looking for nodes left to right, and once you found one, open em in Document Cloud. I don't know, I don't think I got faster. There may have been more density of interesting emails at some point or another. But I mean this is really well displayed when it comes to streamlining looking at documents, does that make sense?
[00:27:09] MB >> Yeah. You would move on from pocket to pocket of densely interesting things and then there would be a lull period of not-as-interesting emails. I suppose.
[00:27:18] JW >> The speed factor, you're talking about just clicking and glancing, it could literally be as fast as 3s per email. Until I got to one that I needed to know, but other than that the speed never really changed, clikcing and reading.
[00:27:40] MB >> So as you're moving through these you mentioned that you're taking notes in Document Cloud as you were finding relevant documents in your 'Important' tag. What other things were you doing as you were working through these emails? What other applications were you using, what were you keeping track of?
[00:27:54] JW >> I was definitely 䯮't know if it would have been smarter to note something as interesting and go back to it later, but I was definitely er �oing to open my 'Weird' tag for this, because whenever I found something interesting that had nothing to do my project or obviously something that had to do with my project I would go ahead and background the person if I didn't know who they were, if I thought they might be interesting.
[00:28:26] JS >> What do mean 'background the person'?
[00:28:28] JW >> Oh, like actually go through several internet sites that I love to use that verify who they are and what they do, find their email, find their phone number, and I'd keep a separate list whenever I'm doing any research of people to contact, what their email is, what their number is, so I can call them later.
[00:28:55] MB >> You mentioned in your notes that the 'Wierd' tag might serve as the basis for some future work, some future stories, right?
[00:29:01] JW >> Yeah, yeah, there's a couple of really weird things that got lopped in. One was a, there's an email chain entitled "Attorney-Client privilege" but they had spelled 'privilege' wrong so that's why it had made it through the legal review. It's just a weird one where there was some officer was having his hard drive seized and they were discussing how to do it, like how to go and seize another officer's hard drive for some investigation that I don't know anything about. So that's definitely something I want to look into. Make sure, you know, that it wasn't child porn, was it - we have a big police corrpution ongoing thing here, was it something to do with that? related to that? Completely unrelated to my story. But very relevant for later. I'll see what else is in there 30:00] JS >> How many, you said that you had backgrounded a bunch of people, how many are we talking about 배:30:10] JW >> Definitely dozens. I mean the records request was a supposed to be a keyword search based on six or seven people's emails, but they were�ow or another I could emails that had nothing to do with any of those six people. So I don't know how they ended up doing the server search but they [City legal clerks] definitely grabbed every email that mentioned 'Panasonic', 'Panasonic U1', 'U1' [all police equipment names], like it was a pretty extensive keyword search they did. Yeah, I think I easily - I'm actually good at backgrounding so it's not a huge process to background someone quick, but I'd say easily dozens.
[00:31:07] MB >> I notice so far that you haven't used the cluster, had interactions with the items plot, the scatter plot. Did you use those features during your exploration or did you stick mostly to the tree view?
[00:31:18] JW >> The only thing that I ended up using this [the items plot] for was 䠉 actually used it a little bit and then quit [hits 'Cluster' button], I used it to, whenever it came down to looking for rogue emails, I could go and zoom in a ridiculous amount click to my heart's content on untagged emails. I was really getting obsessed later on with like being so close to having every email tagged. I wanted to go ahead and get every single one. Obviosuly in the end I couldn't but part of the reason that I knew that I couldn't was because at the beginning I wasn't tagging every single one of them硳 passing up emails, so that's why later on I created the 'Trash' tag. so I could just tag it and move it out of the way. It was a bad idea later on, but it got me closer to all those unread emails.
[00:32:30] JS >> Why was the 'Trash' tag a bad idea later?
[00:32:33] JW >> I should have called it㨯uld have had another tag for it, like some broader tags to begin with, by department, like it could still go under the IT department of the city of Tulsa but not necessarily into my 'Important' tag, so that way I'd know it's not鬬 wouldn't be the trash tag actually because there's a couple that aren't necessarily the trash. Some of them are like brochures or spam email, so yeah that belongs in the trash, but others I was just trying to get out of the way.
[00:33:19] [selects an document in node list] Yeah like this one right here is from an IT person [name], he, I had a lot important emails from him, so to have this one in the trash was a bad idea later on because I may very well want to look into [name] later.
[00:33:47] JS >> So it sounds like the reason that the 'Trash' tag didn't work was you, as you started going through this you got a better sense of what was important that you may not have known to begin with.
[00:33:59] JW >> Definitely, definitely. That and the 'Trash' tag, my process of tagging or my idea of what should be tagged evolved as I went along. Later on when I learned that everything should be tagged that was too late. There were already some things that were mistagged.
[00:34:27] JS >> Why should everything be tagged?
[00:34:29] JW >> That way I don't have to go back and read it, if it's tagged into something then it's either important or not, it's not unknown. From the very beginning, I knew that I didn't want to re-read anything that didn't need to be re-read, but I didn't know exactly how to do that. 堳hould go ahead and take that extra half-second to tag something rather than spending 3 seconds later on 35:07] MB >> You mentioned in your notes you would have preferred to have say a "read"or "touched" tag, a "skimmed" tag, like [indicating that] you've seen this document before, but not necessarily tag it. Somethign that reflects that you've seen the document before, but not giving it a tag.
[00:35:25] JW >> Uh, that sounds great. I think my suggestion was a little different. Was that the one about hot keys?
[00:35:33] JS >> [asks clarification]
[00:35:33] MB >> [flipping through notes] I didn't see that either. What do you think about hot keys?
[00:35:44] JW >> At some point I thought that maybe a䯮't know I had a couple of ideas for hot keys, but maybe your top 5 tags [ॲ flipping noise on my end䥮tify as a hot key, your top 5 or top 2, that way you could have one that is your 'Relevant' hot key and one that is your 'Irrelevant' hot key, and everything else would fall into sub-tags of 䨩nk part of the reason I fell into a little, not trap, but I had to evolve my ideas for what I wanted to tag was that I began my tagging by who was talking and later on like 'City IT support' I was tagging them by their department and then other tags were by some other preconceived notion about what I wanted. So just as a speed thing as soon as I'm looking through an email, the hotkey would save me having to go up hit a button.
[00:36:57] MB >> I think I found you mentioned in your notes that you would have wanted a function to highlight unread documents and have a standard that when you complete a successive broad review of emails you could highlight just the unread ones that you haven't skimmed.
[00:37:15] JW >> Jonathan and I talked about that a little bit. Kinda when it was unecessary 䯮't know if this is a problem, it's a good idea I think. It's a problem that you always run into is that you're in the final step of your review and you have you know, you have 100 emails that just haven't been read or you don't know how to get to, it would be nice if you could just highlight those and just go through them real quick.
[00:37:54] JS >> Yeah I mean I think I'm also applying some changes to have nodes hidden or shown which will speed up the process a bit. But this is really valuable feedback because you know it's not clear to me that you necessarily want to look at every email. It sounds like you were concerned that if you did at least look at every group of documents 38:25] JW >> I think what killed me was right when I thought that I had read everything I at that point was clicking on an unread email and what was its something from the chief of police, who I don't have many emails from at all in this document set for whatever reason. There was an email from the chief of police about traveling to Chicago. It turned out to be nothing, it turned out to be innocent, but when I saw what he was talking about the trip to Chicago, that, all of a sudden emed super important at the time. So unfortunately what happened to me was at the very end I found an unread email and then that was important so I got obsessed with going in and finding the ones that hadn't been tagged and going ahead and tagging them.
[00:39:15] JS >> And did you find anything substantial in the process of cleaning up all of the little ones?
[00:39:23] JW >> I found a couple of importance but nothing actually more important than the tag 'Important', that's all. Does that make sense? It was relevant but no, it didn't change my story at all. So a little bit of good a little bit of bad there.
[00:39:43] MB >> How much time do you think you spent in this cleaning up process, in which you might have found something but didn't have anything that changed your story?
[00:39:53] JW >> It was pretty substantial at the end. That being said, at the end, I'm in like the 'show 1 node' [in the tree view] form which is 40:07] MB >> Really hard to navigate.
[00:40:10] JW >> Well not that, it's by definition, there's a lot, Overview isn't working for you at that point. It is in the sense that it's still organized - wonderful, Jonathan - but it's not doing what it's made to do, which is to point you toward clusters of emails, just looking for whatever you could find.
[00:40:36] JS >> So let me ask you, I mean you mentioned that you didn't have a lot of emails from the chief of police, so you know Document Cloud has a search function, did you ever search for the chief of police's email address on Document Cloud?
[00:40:52] JW >> Oh yes. And you're absolutely right on that, that once I got to the point where I thought I was done in Overview, i definitely went over and used the Document Cloud side to do more of that type of searching . You know, the big difference there is that searching for the chief in Document Cloud is very effective, but searching for Will Dalsing, which is one of the main characters that I have tons of emails from, uh that is not effective at all in Document Cloud. So yes, searching for high-yield key words over on the Document Cloud side was great.
[00:41:39] JS >> Can you talk a little bit more about that process? You know, how you decided what to search for, what you found 41:51] JW >> At that point a lot of the emails had notes on them and so what I would do at that point was printing out all of my notes, looking over all of my relevant emails, and then when I wanted to know more or go and find an email that I knew existed but didn't necessarily know how to find it that's where my keyword searches came in. So I would constantly refresh myself with emails, what I needed to find by going to the keyword search in Document Cloud.
[00:42:29] JS >> And was that, did you sort of go back and forth between the two, or did you - was it sort a search phase then an Overview phase, or how did that work for you?
[00:42:38] JW >> As a process, there wasn't much of one, what ended up happening was there was certain emails that seemed like they should have another email attached to them, or if I wanted to get to an attachment and it wasn't there. So at the very end I was constantly going back oto the outlook file, I was going back to the original PDF that had attachments attached to it, I was going to the Document Cloud version to check on dates and figure out when emails were sent, so there wasn't a process there but definitely some switching back and forth between both programs and the actual directory to actually find emails.
[00:43:35] JS >> Did you ever discover a valuable attachment?
[00:43:43] JW >> Yes, several times. One [example], this kinda goes into the semi-related or 'Weird' file ࢥ relevant for me to check on later, there was an attached photo of all of the old computers, all piled into a wooden crate, like 200 computers all piled into a wooden crate. And one of the officers was complaining that 㲡te of computers that we're doing nothing with. It was kinda a semi-developed side-story.
[00:44:26] Then also attached were the guys in charge of writing the inivitation for bids for the city, they were attaching their word documents that had their early rough draft versions of these, so yes, definitely, there was a bunch of emails that would just reference an attachment that I'd have to go and find, so it would either be on the Document Cloud side of things or in the folder or maybe neither and I'd have to go all the way back to the Outlook file and grab it from there. And then also interestingly there's several attachments that I really wanted and I had to go back to City Legal and ask them if they had stripped them themselves, and I got a strict denial on that, but uh, I still think I got 쩴tle bit.
[00:45:18] MB >> So you never got more than those 300 [attachments referred to in JW's notes] that were initially sent to you, the attachments, the 300?
[00:45:24] JW >> Oh no! I got lots more, I have no idea why - you can't see my face right now but I'm doing the 'bunny-ear' quote thing, I have no idea why they attached those separately or gave them to me separately, because in the end there were all kinds of attachments to the original emails that for the most part I could go back and get, but like I said there were several that were missing, I went back and asked them [City Legal] about them [the missing attachments], and they said 'nope, that was an attachment or whatever that's not attached to the original. So yes, there were many attachments that were - I could later go back and find.
[00:46:13] JS >> Did you feel - this is sort of a very speculative question but, did you feel that you were hindered by not being able to search the text of the attachments themselves as opposed to the text of the emails?
[00:46:33] JW >> No - that might have muddied it up a little bit I think, there's my concern u're searching 䨩nk what you're getting at is that in Document Cloud you would be searching for the attachments, is that right, the attachments as well as the 46:53] JS >> Well in principle you could throw the attachments into documents or PDFs or something and then throw them in there with the emails and so that they would 47:04] JW >> Oh I actually do have, I guess I didn't give you access to it, it's 䨩nk Matt said it, there's like 200 documents in a u do have access to that.
[00:47:19] MB >> Yeah you mentioned in your notes that you first got those initial 300 attached documents and not the 8,000 emails that were coming.
[00:47:27] JW >> Yeah, gotcha. Yeah yeah yeah. So I did have those in Document Cloud but a lot of those were, maybe this is the reason why there were taken out of the original document set is because they were, a lot of them were very relevant, and they were Excel sheets and graphs and PowerPoint slide shows, so they weren't really text-driven, but every single one of them, -er not every single one of them, but a lot of them - I actually needed to go ahead and look through. So no I didn't feel hindered by Document Cloud on that front, I was able to look through most of them and the ones that were spreadsheets or whatever I made a note about what they were about. Is that what you're asking?
[00:48:26] JS >> Yeah I mean this is just for our own curiosity for this type of work, one of the questions we have to look at is how much of the good stuff is in the attachments versus in the emails, obviously it's a difference in how you have to handle. 48:46] JW >> That's interesting. It was nice in that regard to get the attachments two weeks before I got the emails even though I was supposed to get them at the same time, because it gave me a chance to go ahead and look through all the attachments and so on my side that was nice I can see why that would be a problem.
[00:49:05] MB >> So in addition to the attachments, how much, do you think, got lost by just having the text of the emails in Overview?
[00:49:14] JW >> Uh. Say that again?
[00:49:16] MB >> Well, how much data did you think you lost, by just having the text of the emails?
[00:49:26] JW >> I think it was a pro and a con, just stripping it down to basic text, I hope it actually gave Overview a better ability with the algorithms of connecting similar ones because it was just pure text. The con was just the logistics that I couldn't throw up the emails as-is, I had to strip out formatting.
[00:49:58] MB >> But in that you lost metadata, correct? You lost information about each document.
[00:50:07] JW >> Yeah, yeah. That was huge in that I had to go through all the 宠I was creating my timeline of the story, I had to actually go in 䳠of the emails still had dates in the actual text, but for some reason they lost their metadata date, and so it was difficult for some emails to establish when it was sent. When you're talking about� a 100 emails that were very relevant to my timeline, having to go back and find, I'd say 100, having 30 emails that were relevant to my timeline, just difficult to have to go back and find all those. It would have been nice if the metadata was still there, if the date was still attached to every email.
[00:50:56] JS >> So where was the metadata? Where did you get it from?
[00:51:00] JW >> That was one of the problems that I was ෨ere was it? It was on the original PDFs that I made that weren't going on to the Document Cloud well. So I still have a folder full of those so I could using a couple of different things, just the basic Windows folder search or just going in and finding 鮧 through them in alphabetical order, and knowing, like you just said, the email, where that file name should be, going in and looking at the file that way.
[00:51:37] JS >> Yeah I think I'm not quite understanding sort of the eventual conversion and upload process, not sure if now is quite the right time talk about that, but I definitely would want to dig into that with you, in some detail.
[00:51:55] JW >> I know, it falls under the category of "it's always going to be difficult" and I'm guessing one of the more difficult problems you'll run into is [when they come?] in Outlook [format?].
[00:52:09] JS >> Yeah.
[00:52:12] MB >> So Jarrel, you've mentioned that you might be willing to use this [Overview] again for some of your future stories, because of the stuff that you found. But I'm curious as to whether or not there has been any projects in your past that you could have used Overview for.
[00:52:27] JW >> I can think of one huge one. I had requested, years ago I had requested all of the phone call logs, or sorry - dispatch call logs for campus police department at the University of Oklahoma for it was, I think it was a year. I mean you're talking about 200 call log / dispatch logs a day. So looking back, yeah that was years ago, and that was a huge dataset. Other than that, I'm actually 27 and just went from the cops beat to this job about a year ago. And although I've been obsessed with documents before that this is the first position I've had where documents have been, going forward will be a part of my job.
[00:53:30] MB >> Well I'm curious about comparing, part of this conversation I wanted to steer in a direction, was to compare your process now to some of your previous processes on similar projects. So maybe we can use the University of Oklahoma dispatch call logs as an example to compare to, if we can.
[00:53:51] JW >> Yeah, you'll love my process on that. That was a, actually, they printed them out for me because they were like a, they were off a DOS-like computer program or something like that. So they actually printed them out for me and I had a stack oh, 2 feet high or so, just of call log after call log. So what I would do take an inch or half-inch of files and I would go through page-by-page and just quickly highlight the [岠page that I was looking for and the ones that I was looking for were to check on 硳 checking on what they were reporting their numbers were, to make sure they were correct, so I was only looking for certain calls. So I'd go ahead an I'd highlight every call that was relevant and then once I had that inch or so highlighted I would just go though and put them into a spreadsheet, data entry. It was sad too say the biggest waste of time I'd ever done because of course they were reporting their numbers to a T.
[00:55:06] MB >> So there was no story in the end? Or there was a story but it just wasn't the 55:10] JW >> There was a story but I spent like 2 weeks trying to make sure that they were reporting their numbers correctly and they were. So the story in the end was all about campus security and campus police. You know it covered everything from rape to stealing on campuses and trying to discuss that.
[00:55:36] MB >> So in hindsight do you think that you could have - had Overview existed at the time and you could have preprocessed that data and put it into Overview, do you think you could have used Overview for that project? And if not, why?
[00:55:50] JW >> I'm not sure actually. The call logs didn't have a lot of notes or [anything] like that so probably not, it was just whether or not it was a all about what we callలobably not actually. But maybe on on Overview and definitely on Document Cloud. �ot sure actually, I don't know 56:16] MB >> Yeah, it's hypothetical but I'm interested in comparing the process but also specutavely what you could have done, or what you could have used Overview for in your past projects.
[00:56:31] JW >> I haven't thought about that. I can't think of many where I've requested this amount of data. I'm trying to think of somewhere I had less that I could have used something on it.
[00:56:45] MB >> So conversely, do you think you could have done this story [the Tulsa PD story] without Overview?
[00:56:53] JW >> Yes only because I would have, er, there wouldn't have been a certain amount of journalistic joy I would have from spending like 4 months of my off time looking through email after email. That being said, it might have been a bit of a sand pit that I couldn't crawl out of because, you know, 4 months, 4, 5, or 6 months, after you've read an email, will you really remember it? Did you put it in the right file to look at it later? I don't know. Doing this in 2 weeks in ridiculous.
[00:57:27] MB >> Do you think that there's a threshold then of how much or how big a dataset would be using it [Overview], using Overview, or just going through by brute force, reading one by one?
[00:57:39] JW >> I have an example of another reporter that just requested, what was it, oh it was customer complaints from our local emergency medical service, and they give her a box full of complaints from the past 2-3 years, and there was about 300 or so. A lot of them typed, a good chunk of them were handwritten, but full-on typed. I still think that she hasn't looked through all of them but I told her if she needs a decision on whether she wanted spot check some and keep it simple or if she wanted to go ahead and look through them all and focus on those documents. In which case I told her about oh there must have been 300, 400 complaints that Document Cloud -er Overview could help her do that. And if they were in paper format it would just be about digitizing 䨩nk the biggest part of that process would be digitizing them, sitting there at a scanner for a day or two.
[00:58:54] MB >> OK. So have you shown Overview to some of your colleagues or editors then, and has there been expressed interest?
[00:59:03] JW >> Yes, It's still difficult to define a project that would work well on it, to describe it to the person. The matter at hand is how to making them make the decision of "do you want to do it this way or do you want to do it that way?" If you want to do it this way [using Overview?], then it's really simple and I can show you how to do it. Does that make sense? But it's going to require more time than the other way, which is to grab a handful and look at em and use em. I'm trying to put the question to them: "do you want to look at them all? are they all relevant to you? or do you just want to use a few of them? In which case, go ahead, you don't need Overview."
[00:59:55] MB >> OK. I'm also curious just to find out a bit more about your background and your context. Before Jonathan joined this conversation, you showed me that you're in a I would say a mid-size but fairly busy workspace. So you have a lot going on around you. How much of this process as you going through and working through these emails did you ever seek collaboration with other people, did you ever have anyone joining in on you on spot checking any of these emails or reading any during the process or was this a solely solo endeavor?
[01:00:38] JW >> This was definitely solo, except for - we have a great data guy that was, in the beginning he was helping me, before I had Jonathan's help with doing the pipeline that made the CSV and upload to Document Cloud at the same time, I had him [the data guy] helping me on how to create the CSV with all the PDFs. So ౺01:09] MB >> During your exploration and analysis stage it was백:01:14] JW >> No. Solo.
[01:01:17] MB >> So I also am curious about the time frame for this project compare to some of your past projects. It seemed like it was several weeks of analysis and a couple of weeks to write it up, right? How does this compare to some of past stories?
[01:01:34] JW >> This is the biggest project that I've done in my early career. There was early on, I was still keeping up bit of a workload of other 䠬ater on towards the last 2 weeks of it, it was just this project, getting the writing phase done.
[01:02:02] MB >> How long would a typical story take, then, usually?
[01:02:08] JW >> A usual project, if it goes into the project category, I would say, at most about a month, around here.
[01:02:23] MB >> So this is definitely longer.
[01:02:25] JW >> Yeah, yeah. This is definitely 렬onger than any other.
[01:02:35] MB >> So you mentioned that you're early in your career. I'm curious about your background, how you got into doing this type of work and knowing that you could do this with this much data.
[01:02:49] JW >> Let's see, I've been on the projects team for coming up on 2 years, and a big part of what I've done is stuff just like this, kinda working with technology to 쯴 of the things we do around here. So I'd say that's one of my computer strengths. I'm a pretty good reporter, I'm an OK writer, but I'm definitely good about bringing into technology into doing things, a little faster than colleagues. That would be strength I hope, anyway. I'm in the middle of the newsroom, by the way, so 礠rather [chat] myself out without [chatting] myself out.
[01:03:41] MB >> What were you doing before 2 years ago?
[01:03:45] JW >> Cops beat. I've been doing the cops beat here for 2 or 3 years before that.
[01:03:58] MB >> So what did that involve?
[01:04:03] JW >> That involved basically competing with TV, we had, at the time we had 3 cops reporters, one was morning breaking news, the other one was late breaking news, and the third was the sorta cop reporter, like their actual beat was supposed to be doing the bigger cop stories. So for the 1st year or so I was on the night beat, and the next year ೯, the morning beat, and then I transitioned briefly over doing the cops main beat before getting this position⯵gh begging and pleading.
[01:04:50] MB >> And you mentioned just a few minutes ago that you have a background as a computer programmer as well [misheard].
[01:04:55] [JW exits screen sharing, video resumes]
[01:04:56] JW >> No no, my � geek, I wanted to to be a programmer but I'm not nearly cool enough. The best I could do is take something that Joanthan gives me and then Google the proper language of something I want to change, and then trial and error over about 4 weeks to make a simple change.
[01:05:28] MB >> So is that what you found yourself doing during this project, relying partially on Jonathan, partially on Google, and partially maybe on this IT guy in your department.
[01:05:38] JW >> Obviously Jonathan and Ted Han [Document Cloud guy] did some things for me that were absolutely not on Google, and not something I have any idea of how to mess with. Then like cleaning the PDFs and renaming, simple scripts like that, I found versions on the internet and augmented them to my needs. Real simple, like 15-line things.
[01:06:08] MB >> I think you mentioned at one point that you used an application, a program online that would work with Outlook and would directly export each email as an OCR'd PDF. What was that?
[01:06:22] JW >> I think I actually, there was a trial version of 䯮't remember the name of it right now. I looked through a couple of different things that Outlook would do. There was a trial version of a paid program that let me do 75 emails at a time, to convert to PDF, for free. And after spending hours doing all my emails that way the first time, the second time around I ended up spending 70$ bucks on the program. I wasn't willing to do it again. But I don't remember the name of it, something simple like 'PDF Exporter'.
[01:07:06] MB >> Also some of your process came through in your notes that were great. Jonathan mentioned that you might be writing a blog post about this. That we might see more of this.
[01:07:20] JW >> Yeah I could, I'm not much of a blogger but I could put something together౺07:30] JS >> I've already had requests from some of people, both here within the AP but also other news hacker types. I sent out an announcement for stories and ᴠyou did with it, almost the first thing I got back was, you know, "what's the details, how did he do it?" And so a bunch of people are interested in the story behind the story, so to speak.
[01:07:53] JW >> Yeah we can talk about it, we can talk about it later.
[01:08:00] MB >> One thing I was curious about was how much process was going to come through in your eventual story. Some of it did. I was also curious as to what extent the readers would pick up on your process, and I was looking through all the reader comments yesterday too and there was this one guy, or one person, what's his alias there, 'Tough but Fair'.
[01:08:23] JW >> I did some background, it's actually the mother of a cop. It's his mother's username, to give some insight.
[01:08:33] MB >> Because based on I think the 50 or so comments so far, that was the only user who went through and started reading through all those emails that you posted. And what was interesting I found the reader changed their opinion after reading all the emails. And I'm not sure as to what extent other readers read through any or some of the emails at all. I was wondering what you thought about that. How ౺09:03] JW >> I was blown away by the comments because we don't write a story that doesn't have 20 commenters tearing it apart, tearing it apart: either the writer, the focus of the article, or some person in the article, over the Tulsa World as a whole. So to have an article where people are like, I don't know, praising the police, praising me, praising everybody. That was unique because you could, given all the documents I think people could just go ahead and read into them any way they wanted which I tought was fantastic, and very strange. Usually they accuse us of taking a side, but how do we take a side if we present you with everything that we used, and they didn't want to attack the police, I don't know it was just very strange to see the way that the commenters were very positive for the first time ever. They felt empowered, I think.
[01:10:17] MB >> Yeah. Did you think it would have been any different if your agency agreed to share all the documents?
[01:10:25] JW >> Yeah, I think it would have been maginified even more, for instance I think that guy's [commenter Tough but Fair] main gripe toward me was that I think if I remember correctly that I was picking and choosing emails to suit my story, and I thought that was great. I was like "hey, I'd love to give you more than 125 emails". So I mean, yeah, I think the more the better.
[01:10:54] MB >> Yeah and that was reflected in your notes too, that it wasn't your decision to share all of them.
[01:11:01] JW >> It wasn't my decision. I think there was some pros to not sharing them all. One of which is that I can take more responsibility for all those emails that we did show. I actually went through and redacted every cell phone, nothing beyond that but I redacted every cell phone. And then every note is very pretty and "our style".
[01:11:29] MB >> Consistent, yeah. I noticed that.
[01:11:30] JW >> Yeah, it's consistent. Instead of going back through and you know, all the emails that I'm like, "checking this out", "check more about this later", I didn't have to go back and delete that note or whatever, augment it. So if we're only showing 125 documents, a pro there was that I was able to comment freely, note them freely on the Document Cloud side of things, and later on clean them all up.
[01:12:09] MB >> What did your fellow colleagues think of the story? and your editors?
[01:12:13] JW >> They all said it was very cool and were very impressed, and they want to used Document Cloud in the future as long as it 粥 trying to figure out now whether it's going to be around in 5 years, if we can be assured of that. If so then we might start using it more, but basically it was nothign but compliments, it's something that we haven't done around here, and something that's impossible to do when, what we normally do it host the 2, 3, sometimes upwards of 6 or 7 documents, but it's impossible for us toयn't have a way to show 20 documents, let alone 125. So it was very cool, very new around here and people appreciated it.
[01:13:09] MB >> Were people [colleagues, editors] curious about how you explored the documents?
[01:13:14] JW >> Yes, but on the same note, beyond showing them what Overview looked like and showing them it did in fact organize my emails, they weren't further�ort of the interested party. I have feeling that if we we use it I'm going to be the liason in the process. I'll set it up on their computer and all that stuff. Other than them getting individually acquainted with it.
[01:13:49] MB >> Right, go through this process again.
[01:13:49] JS >> So let's talk about that for a minute if we can. So I mean, I understand, like you sayᴧs the岥 do we start in terms of making it easier for the less technically-inclined to do this type of work?
[01:14:10] JW >> I think like you guys are already working on making it an online application is huge. One idea I had was to make it have a section of eventual website that says "here's the姳 our brochure, our 4-step process or 3-step process" and then that, I think that set people off on the right direction. Now, how does it become more user-friendly? I'd say Photoshop's not user-friendly but once you learn how to use it it's invaluable, so I don't know, I'm sure there's a very interesting problem you have ahead of you of making it usable while still making it do what you need it to do, and so people need to learn it.
[01:15:14] MB >> An unavoidable learning curve.
[01:15:16] JS >> Yeah, I mean, that's the art of software design in a nutshell. How about sort of conceptually? Like, were you ever wondering why it categorized documents together, did you have to explain it someone else? Were the algorithms clear or magic to you?
[01:15:34] JW >> Well a little bit of magic. I think right or wrong, I kept telling people that it analyses bi-堭issed the word all of a suddenਲases, is that the right word? Bigrams, right, there you go - I had it before but not now. - it analyses bigrams and other groups of words, compares them in frequency and in other categories to each other and to similar emails and them groups them and organizes them. That's kinda my spiel that I gave to anyone that was willing to listen. It made me feel cool and made them maybe feel like I was explaining it. Now, I don't㴩ll don't know if I'm right or wrong there, because there's a little bit of a magic factor.
[01:16:34] MB >> Did you teach yourself that ౺16:39] JS >> 堹ou everⲹ⥠you ever confused or surprised by the way that it had grouped things together?
[01:16:46] JW >> Let's see. Let's go back to the screen share. [screen share resumes]. 糠look at it this way: Q1 training classes [selects a tag, purple nodes highlighted in tree, documents highlighted in items plot]. That's the tag for all the emails regarding training and it very well grouped them in most places, like here's a whole entire section about training classes for the computer. And then it grouped them very well, but also left to right, they're all over the place. So there was, that's a good example of I don't know why it was that way. And some of the emails like the service desk were very well grouped again [selects service desk / turquoise tag], but also these were so similar that I guess I never did why they weren't very well put together. With that being said, or like I said, they are quite well-grouped together, so I don't know, I was both very happy and not, very happy and a little confused.
[01:18:03] JS >> That's interesting, thanks. Sorry, Matt. I'll let you get in your question.
[01:18:09] MB >> Oh I was curious as to how this conceptual understanding, was it rolled into that initiakl 2-day period, in which you were first exploring the documents, or did you think this sort of built up over time, you were able to communicate conceptually.
[01:18:23] JW >> I feel like the two day period was篴, I glanced over all of them, and it seemed that there was祮t from not understanding to understanding a little bit about it. And then as it went from two days to two weeks I had more questions about why did group絥ss. It shouldn't perfectly work, but, some of those questions did come up.
[01:18:59] MB >> Alright, I think there's a few other things, a couple of things I'd like to ask. What do you think is unaddressed with Overview, like what problems are still remaining for you, for this type of project?
[01:19:12] JW >> Uh, say that again?
[01:19:16] MB >> What problems are left unaddressed?
[01:19:22] JW >> [pause]
[01:19:27] Let's see, what haven't we addressed౺19:29] MB >> And is there just something you still can't do that you wish you could have done when you started this project?
[01:19:36] JW >> Again my idea about looking at the unread emails as I thought of, as a final step thing. But in the in-betweenᵳe]
[01:19:53] I guess there's a堦irst thing that pops into mind as I think about it is a more modular design that I can customize the UI a little bit, to my needs. Another one would be౺20:11] MB >> So how would you do that?
[01:20:14] JW >> Uh that's up to you guys. I don't know if Java has a good job of allowing it. But maybe the browser view could, you could widen that on my square screen for instance. Like I said, whenever I spit it out to widescreen天o interruption on JW's end]. Whenever I brought it out to widescreen it was very well displayed. It's just that early on, I noticed, or I thought I noticed that there was display problems and that's where that idea came from, was that it could be moreᴠyou could modify it yourself. That being said, when I went into widescren mode, it's like "I don't really need to make any modifications", it's well laid out. [pause]
[01:21:26] Oh the tags [feed]. Whenever you loaded it kinda randomly loads the tags, maybe it's not random. I'm not sure actually, but where the tags were would change.
[01:21:42] MB >> Relative to your last use?
[01:21:44] JW >> Yep. And I didn't know if for usability purposes if they load in the same position or౺21:57] JS >> Yeah. That sucks. That's a good one. I hadn't even noticed that. But that does strike me as an issue.
[01:22:05] JW >> The only place where it was an issue was you spend like 4 hours looking through emails one day, you have kinda a system of: read-click, read-click, read-click, and the next day maybe my 'Important' would be below the scrollbar, and you add a new tag automatically goes back to the top of the scrollbar, so I had, before I started Overview堧et where I'm going. It was just kinda a small thing. [pause] what else?
[01:22:45] Oh I just thought of a great one for you, kinda as an end product would be to 岹 section should have sorta a little question mark to tell you what it does. Like we discussed about how I use the [folder] section. I wasn't sure what it was doing for me, so I didn't know how to use it.
[01:23:17] MB >> And you still don'tems like you still don't really get the sense of how you're going to use that folder section in the future?
[01:23:27] JW >> Yeah, I mean I found a couple of used for it, I know it does something, but I don't know what I should be using it for, definitely I still don't, you're correct.
[01:23:40] MB >> I think that's about it, I have several pages of notes here in addtion to yours that are really great. I think this has been a really productive talk. Is there anything that either of you want to ask more about what we've talked about?
[01:24:01] JS >> I just had a couple of questions about your background actually, Jarrel. So you didn't do, like, computer science or programming or something, in school.
[01:24:10] JW >> No. Just this last, I'd say, 2 years, I've got a personal interest that in it that I keep [鬬 work in my journalism career. But computer science, absiolutely not. This all started with Doom 2 in 1994.
[01:24:30] JS >> Is there anyone else in your newsroom who is let's say, technically inclined, in this way.
[01:24:38] JW >> Yes and no. There's severl people that have fortᠣouple people are my SQL go-tos. We have another guy who's actually just moved, and was a big loss for me, but he was actually a programmer in the newsroom. His job was to program things on the website and I could always convince him to program something for me on my reporting side. Other than them, there's one or two people on the website kinda like me, they know the basics of programming and what it is. Not necessarily how to do it. So between use we can find a couple answers but that's about it.
[01:25:27] JS >> Alright, thanks.
[01:25:34] MB >> Anything else to add, Jarrel, about your experience? What you're going to do next?
[01:25:41] JW >> I'm incredibly happy with it. I feel like I can't go back to the City with an open records request for like a year, but, I'm defintely looking for another楠kind of put the word out that if you have an opportunity, that other people have the opportunity on their beats to grab a batch of emails, do it. We got theਡve the tools to operate and use them. So I've been incredibly happy with it. The other nice thing is knowing that the next time around I can actually衶e a lot of my problems worked through and the next time around I can actually establish a process that works from beginning to end, instead of having an idea of a process and changing it, and changing it, and changing it, over and over.
[01:26:32] MB >> Yeah, and even since in past few weeks, there's been some more scripts to help work with Document Cloud, the OverCloud script, the DocSplit utility.
[01:26:40] JW >> Yeah, Jonathan told me about that an dI haven't actually tried em out. I need to keep familiarizing myself or else I'll have to sorta do the next time. Hopefully by that time, it'll be out and about, and we'll be good to go on it.
[01:26:59] MB >> And you might be using that 'Weird' tag to examine a few stories in the future.
[01:27:06] JW >> Oh yeah, I've already堧Weird' tag actually, a bunch of them, might just print it out and hand it out to other people who, they belong堫now, belongs on their beat.
[01:27:18] JS >> Oh actually that's a question that I definitely had. Yeah, what did you find that you weren't looking for?
[01:27:28] JW >> Oh, that I wasn't looking for. I mean, I found kinda lame but apparently they're trying get the city to buy a new building to put their police department in, or at least that was an idea that supposed. An idea that someone referenced like six months ago. You know, our city's poor so we're all basically in debt, just like every city right now. That idea might have fizzled, but it was never in the news, so that's a big one.
[01:27:57] Another one was a supervisor telling an officer that she had a sexy voice.
[01:28:10] Another one was, I discussed it earlier, was a police officer's hard drive being seized. I have no idea why they were衤n't figured out why they were seizing it.
[01:28:24] Another fun one was an officer was, they were having trouble because of this computer problem, system, they were having - someone was having trouble removing a police report from the system. And the story goes that an officer had filed a police report against security officer that got in a wreck, and then the security company came to that person's supervisor and asked to take the report away, or whatever, change the report. So the supervisor tried to take the report off the system but couldn't, so basically it sounds like they're choosing after a report's been written to get rid of it, to do someone a favour. There's a couple real nice gems just like that, that don't belong in my story but I came across em and know have em. And I'll go back through them later.
[01:29:26] MB >> You wouldn't have found those had you not been exploring very broadly in Overview.
[01:29:31] JW >> Not a chance. Even if I had gone through a bunch of the keywords, I would not have come across those.
[01:29:38] MB >> Short of taking your 4 months to read all 6,000 documents.
[01:29:43] JW >> Exactly, yeah.
[01:29:46] JS >> Why do you think it would take 4 months? Where does that 砤o you come to that estimate?
[01:29:51] JW >> Maybe longer, maybe shorter. I have no frame of reference for that. I don't ever want to read 6,000 emails, although I would. [laughs].
[01:30:07] JS >> Yeah, programmers have a similar OCD thing.
[01:30:10] JW >> Yeah, yeah, yeah. Exactly. It's fun.
[01:30:12] JS >> Sort of, yeah.⩧ht. Alright cool. Let me justah, I guess this is in the logs but I might as well ask you: did you ever ever use then feature where you open up the document in an external browser백:30:32] JW >> Oh yeah.
[01:30:34] MB >> Yeah, it's in the logs.
[01:30:35] JW >> Especially when I was needing to track down an attachment or track down a date that didn't get into the final formatting in my documents, I would use that to kind of back track to the original folder, the original documents, or just to go to the [ could directly add a public note, for instance.
[01:31:02] JS >> Yeah, did you do that while reading at all, like was itut to get a better view of something interesting or백:31:10] JW >> Yeah, some of them, the way its formatted, most of the emails look great in the Overview UI, but sometimed for whatever reason it would be like a really wide document, like something about the original formatting would screw up the way it exported to PDF, and so it would be completely unreadable in Overview but you just hit Enter and there you go. So yeah, absolutely. Or the thing I love about hte browser that I found out later was it functions pretty well as a browser, I could click on the original document PDF and actually brings it up right there, so yes, I used it also I needed to use it sometimes just by bringing up the original PDF.
[01:32:03] JS >> Cool. Alright. Well thank you, Jarrel, I must say you've been an incredibly tolerant and capable guinea pig. I don't know many users who would be willing to put up with the amount of crap that you put up with, to get it running for you.
[01:32:21] JW >> Well I've enjoyed the hell out of it and appreciated you guys getting me on the right track so many times. I hope also that it'll justify all the work that you've done because I really do appreciate it, very very useful on my end.
[01:32:36] MB >> And thanks for letting me share in this studying this process, it will definitely help with my future research when we talk to more Overview users but also more broadly about any type of visualization that supports exploration and discovery of things that you might not know that you're looking for. This is great.
[01:32:57] JW >> Absolutely. Please make me aware of any big changes as they come along. I mean, I need to, I want to keep in touch with the Overview project, not just the next time that I have 8,000 emails sitting around.
[01:33:16] JS >> Yeah, we're easy to find.
[01:33:18] JW >> Will do.
[01:33:21] JS >> Alright, thanks, Jarrel.
[01:33:22] MB >> Yeah, thanks, Jarrel. Thanks, Jonathan.
[01:33:24] JS >> Yep. I'll talk to you guys later. Bye all.
[01:33:27] MB >> Yep. Thanks. Bye.
[call end].
interview date: 13.02.04
story link: [[Own a Gun? Tell Us Why|http://www.thedailybeast.com/articles/2012/12/22/the-best-of-our-gun-debate-readers-weigh-in-on-owning-firearms.html]]
MHK [[blog post about his process|http://newsbeastlabs.tumblr.com/post/42928650674/the-other-month-we-asked-readers-why-they-did-or]] (02/12/13)
[00:01:15.28] two CSV files (~800 and ~500 quotes, from gun owners and non-gun owners respectively) 1 quote = 1 document
[00:01:40.06] comment: Overview vertical dividing bar, unable to move/drag/resize
[00:02:41.13] comment: lack of keyboard navigation for tree view
[00:03:19.29] no tagging used, short turnaround, simple clustering, no plan to revisit data
[00:03:49.24] identified a "military" cluster in the tree
[00:04:24.08] back and forth with Excel and its full text search on related terms (navy, army), looking for related items not clustered together in Overview
[00:04:44.06] similar approach with "suicide" cluster; verified clusters in Excel, not out of distrust of Overview's clustering, but as a guide for things it might have left out, searching for thematic similarities, searching other key words from the theme in his head
[00:05:54.20] Q: themes known a priori or emergent? A: emergent from the data, jumping out at him such as the cluster around "needs", this theme framed his reading of the "gun ownership" group
[00:06:38.23] mislabeled the document sets (own / don't own)
[00:06:54.13] discovery of themes in one csv (don't own) would lead to parallel analysis/search of same theme in the other csv (own), though it wasn't explicitly clustered in that document set, those talking about needs wouldn't explicilty mention "needs"
[00:07:49.10] Q: depth vs breadth exploration? A: depth-first until all documents were farily similar
[00:08:26.04] Q: ideal cluster size? A: looking for clusters that looked like they were on a topic, size of clusters didn't matter, as long as the theme wasn't general or already examined (need vs. don't need)
[00:09:24.06] "and for our purposes, it's kind of a distillation, well I guess it will always be a distillation if you're going to use this tool, because you have thousands [of documents], but we didn't need, like, 50 documents, you know, 24 was even more than we were going to publish, so we just kinda needed to you know, get a few that were representatve, so 10 to 24 seemed like a good size then."
[00:09:59.08] examining the "other" clusters, to see if there was anything that the [Overview clustering] algorithm didn't pick up, but touched on some sentiment that was interesting, or an outlier that could be worth writing about
[00:10:30.04] Q: how many documents are considered outliers? A: a good question, quite a few documents were clustered into "other", about 1/3 of documents (~200) in the "don't own" document set.
[00:11:44.14] in the other category, several quotes were 2nd amendment related, then searched in Excel 2nd amendment-related terms that Overview might not have clustered together
[00:12:26.12] Q: how many documents read? A: a good question, read through a bunch before using Overview, in Excel, "Yeah I'd say I read through a fair amount, I can't say if I read all of them, I probably didn't read all of them because that's why I used this tool."
[00:13:21.08] Q: How long did everything take? A: about a day. Q: Was this expected? A: Yes.
[00:13:38.27] Q: Why not tags? Size of document set? A: "It was more about 楬t I was navigating alright without it, and it wasn't a long term project, so I knew it wasn't something that I had to be super-organized about, 률if you were programming something, it's not going to be the most elegant structure if it's for a deadline. I could have gone through and added those things and done the colours and now I could look back and see them, but as I was going through it I was constructing the story and rearranging them in just a text file of which ones fell under what and thinking about how they work together, and that was fine for what the story required." Separate textfile is offline tagging, a notepad for constructing the story.
[00:14:42.24] Q: Is there a point when you would use tags? A threshold length of project or size dataset? A: Definitely. This [project] was such a quick turnaround, starting on Monday [after Newtown, Dec 17], built comment system, published Monday afternoon, gave it 3 days to collect comments, analysis (and writing) took a day, article was published later that week, Thursday or Friday [Dec 20-21]. All while handling another project in parallel. It was a tight week, and it wasn't an expected project.
[00:15:51.22] benefit of having structured data format.
[00:16:34.04] Q: Collaborative or solo analysis? A: Solo.
[00:16:40.11] Q: Reliance on Overview documentation? Blog posts? A: Had previously used the command line previous version, watched JS's video, played around with that. This is first experience with the web interface which is so much simpler.
[00:17:10.29] Q: Why not use desktop version? A: web interface simpler, command line setup a nuiscance, package upgrades and installation an issue (e.g. Ruby), working from home vs. at the office
[00:17:56.01] hypothetically if the desktop version was ready to go (installation, data ingestion issues aside), what about a comparison based on interface alone? Would the process of analysis be different?
[00:19:06.19] re: Overview desktop's linked viusalizations and outlier search support: "I like that, I would try that out."
[00:19:31.24] Q: conceptual understanding of clustering algorithm? A: Had heard JS talk about it, might not be able to explain it himself. It looks for co-occurrence, right? MK sat in on JS' classes where JS discusses different clustering algorithms, so he had that background with it. "After hearing [JS] talk about it a number of times, I knew vaguely what it was doing. But I also knew enough that the limitations." Example of documents citing *different* parts or phrases of the 2nd amendment, but these documents don't co-occur in a cluster relating to the 2nd amendment. To find this broader category MK would have had to do find/search for this content manually. This understanding allows MK to recognize when it's useful to use Overview's clustering and when it's better to do your own searches. "Or my own searches might be totally useless and it [Overview] is much smarter than I am and by introducing my own analog searching through it it's kinda breaking down the more meaningful clusters that it's found."
[00:21:07.29] re: repeating this analysis/search process twice for both datasets (own / don't own), comparing the cluster breakdown between the two document sets: it would have been interesting to have put all of the responses together.
[00:21:51.14] *switches to "don't own"dataset*
[00:21:51.14] MK play around with some of the tags a little bit, but wasn't pervasive or perceived as successful.
[00:22:09.12] Q: did "need" appear in clusters in both document sets? A: this was more of a sentiment comparison rather than a comparison of indiviuals' needs.
[00:22:30.28] "I guess if someone didn't really understand how the algorithm worked based on vocabulary, then they would expect it maybe to be why sentiment analysis. Knowing how it works is useful is then useful because you know why it's finding the things that it's finding."
[00:22:49.08] Q: Could you have done this project without Overview? A: busy during week of Dec 17. Initially the editor assigned the story to another reporter (Matthew DeLuca: "[[Readers Weigh In After Newtown Shooting: Why Own a Gun?|http://www.thedailybeast.com/articles/2012/12/19/readers-weigh-in-after-newtown-shooting-why-own-a-gun.html]]", The Daily Beast 12/19/12). He (Matthew) read (most?) user comments (> 1000) linearly, without Overview. MK thought this approach to be more valid, but not as quick as his Overview approach.
[00:24:17.21] Note: Read [[Matthew DeLuca story|http://www.thedailybeast.com/articles/2012/12/19/readers-weigh-in-after-newtown-shooting-why-own-a-gun.html]], compare to MK's story to determine if the same sentiment and themes appear in both stories.
[00:24:43.15] Q: How long did MDL take to read >1000 comments? A: thinks he spent about a day.
[00:24:51.17] Q: What did it take to convince editor that it was worth the time/effort of using Overview to analyze user comments? A: No it was fine, they were open to it. "What I really wanted to do that I thought that the format lacked a bit was 嬬s up his story] ᮴ed to do this format, I wanted to get it out of the way and let the readers talk about it. You know, short intro, quote quote quote quote. I was just kinda lost, not to totally denegrate.
[00:25:43.24] MDL's story picks up on the "need" theme from MK's analysis.
[00:25:54.12] Q: Did the 2nd amendment debate come up in MDL's story as well? A: No.
[00:26:08.01] Q: What would you have liked to do for this story that Overview didn't allow? Perhaps with other tools, or with more time? Or layers of information that Overview can't provide? A: Hadn't worked with much NLP before this project, doesn't have much of a comparison to. The interface issues (discussed above at [00:01:40.06], [00:02:41.13]). Recalls that scatterplot in desktop version was really useful.
[00:26:53.13] Q: choropleth map in the article. Generated in Fusion Tables? A: No, different service but similar. Used CartoDB. It's their own server.
[00:27:06.17] "How would I be able to associate, like look for, or, you know, cluster them [reader comments] geographically?" Structured (geographic data) and unstructured data (text). Structured data as a colour? That might be hard (because they would appear throughout the dataset). The ability to see some metadata would be helpful. All Overview sees is one column and the text (text, title, and url). If Overview could take into account other metadata in the clustering, that would be interesting but obviously much more complex.
[00:28:33.07] Examining responses by state was facilitated by the CartoDB application embedded into the story itself, in which you can filter responses by state. This structured information is not handled by Overview.
[00:28:56.02] Q: Did you ever attempt a state-specific analysis or pre-filtered responses using Overview? A: No. That would have been interesting.
[00:29:21.22] Also examined state responses in a non-normalized bar chart. Considered normalizing against traffic sources from that page, not totally scientific but perhaps could be interesting. This might have required help from another analytics specialist. But there was no time for this.
[00:29:56.23] Q: Is calling in an analytics specialist part of the usual workflow? A: No. That's never panned out so far.
[00:30:10.03] Q: Would you use Overview again? Or are you using now? A: Also used it for an abortion debate story about Roe v. Wade that immediately followed the Newtown gun debate story: [[Interactive: The Geography of Abortion Access|http://www.thedailybeast.com/articles/2013/01/22/interactive-map-america-s-abortion-clinics.html]]. They asked readers to tell a longer story (than that of readers in the gun debate story). They went through manually and picked the best ones: [[Daily Beast Readers Share Their Stories About Abortion|http://www.thedailybeast.com/articles/2013/01/22/daily-beast-readers-share-their-stories-about-abortion.html]]. MK used Overview for an interactive story where they asked readers if they support/oppose legal abortion [[Readers Weigh In On the Pro-Choice and Pro-Life Labels|http://www.thedailybeast.com/articles/2013/01/22/interactive-readers-share-their-views-on-the-pro-choice-and-pro-life-labels.html]], "we were less thrilled with that reader interaction project because the responses weren't as interesting, so even reading through a bunchs kinda just similar talking points." Briefly ran it through Overview and didn't find anything beyond that.
[00:31:46.12] [switches back to Overview, loads abortion debate dataset]. Nothing jumps out. Q: so despite the 4 or 5 mid-size clusters generated by Overview, the stories weren't as distinct? A: Yeah. Similar talking points in all clusters in this debate. It wasn't clear that any individual opinions were worth doing a story about. The more interesting stories came from the longer-form reader responses from the [[earlier story|http://www.thedailybeast.com/articles/2013/01/22/daily-beast-readers-share-their-stories-about-abortion.html]], which were read through manually. Their goal with these reader responses was to bring have the abortion conversation on different terms ("I'm pro-life, butﴻ, "I'm pro-choice, butﴻ), but readers didn't break out of those general talking points. Didn't need to use Overview for this, just read through 60 responses and that was easy enough to categorize on his own.
[00:33:48.04] Q: Had there been more responses would it have been appropriate to use Overview? A: Yeah.
[00:33:56.09] Q: Any stories from MK's previous work that could have been appropriate to use Overview? A: "I wish I came across troves of documents like that, it would be interesting enough, more frequently." References the AP's stories on the Syria Assad emails (Daily Beast didn't have access to these) - that would have been a good use case.
[00:34:30.03] The idea to use [Overview] on user-submitted content came from [[Blair Hickman|http://www.propublica.org/site/author/blair_hickman]], social media editor at ProPublica, who was interested in doing NLP on readers' opinions. The Newtown story came out a month later. "That's how we came to use it on something that isn't your typical investigate type trove of information."
[00:36:10.21] "My thinking would be that because they [the reader responses] are shorter, you know, 300 character or less responses, that the [Overview] algorithm would have a hard time of categorizing them. Or the counterargument is that people [readers] are forced into brevity so they would resort to keywords and the same more succinct keywords." (the later abortion debate story also initially had a 300 char limit, which was later relaxed to allow for longer responses). Not all reader submissions used the 300 char limit, could this have made a difference? If the responses are too short, could Overview have had enough to go on? Could it work on short tweets?
[00:38:10.21] "Would it make a difference how big the text of the individual document is if it's just looking for䧳 say that what you really want is a 2-3 word string, and if that shows up in a bunch of tweets, then it doesn't matter if the parent document was 100 characters or୩ght actually be better if it was shorter because then that string would be easier to find, it's not buried in other words."
[00:39:16.16] On how Overview clusters key words and if MK would have lumped the two document sets together (own / don't own), "[Overview] would cluster those as being similar".
[00:39:51.02] Note: combine 2 datasets in Overview, see if Overview naturally separates out own/don't own clusters.
[00:40:03.12] Comment: Overview could allow for a configurable "don't throw these words out" list for ensuring that some terms are custom-weighted in the clustering.
[00:40:26.21] background questions.
[00:40:46.14] MK formerly did research in neuropsychology. In college, a comparative literature and psychology major, studied French lit, American lit, (e.g. Indochina war and semi-autobiographical lit on that); cognitive neuropsychology - two fields that discuss "how you make meaning out of stuff", either a text (comparative lit) or its different inputs (physiological activity in the brain), philosophically dealing with similar issues. Not such an odd pairing as it first sounds.After graduation, a parttime job teaching English in Marseille and a intern medical researcher, working on a neuropsych study treating fMRI data, which was pretty boring, checking which parts of the brain were activated, getting coordinates and cross-referencing with a book. He wondered, "couldn't this be scripted? aren't their programs that do this?" did that for a year, but wasn't interested in such a slow track of research, and at the time was applying to journalism schools, law schools. Decided to go to journalism school (in NY).
[00:43:23.18] Q: Computational journalism background? A: took a course in Flash. "I guess that's like taking a course in Madlibs if you want to be a novelist" (only course offered at that time, now improved). After this, getting into programming put MK in touch with people like JS: "And if you're interested in doing stories differently, which is how I kinda see it, whether that means visually or using these types of data programs to mine stories even if you produce just a print story, it kinda puts you in touch with that world."
[00:44:55.27] self-taught in most programming, first forays into NLP. A lot of this on http://newsbeastlabs.tumblr.com/
[00:45:13.05] Q: how often do stories like this gun debate story come up? Will you do this again? A: people were really happy with this project. Definitely. I'd love to get email responses from readers if the topic were compelling enough.
[00:46:00.23] Q: What else is on your data analysis / processing / vis generating tool stack? A: I use R, RStudio, Excel are the only data things, some python stuff, so that. I think I should learn more Ruby. In terms of programs, I like to do a network visualization, which I use Gephi for, and [[jsPlumb|http://jsplumbtoolkit.com/jquery/demo.html]], .js library for doing network vis., which was alright. Programs: [[CartoDB|http://cartodb.com/]] for geospatial vis, previously Fusion Tables and/or Albert Sun's [[gmap features|https://github.com/albertsun/gmap-features]] for vector maps on top of Google Maps, allowing for hover states (better than Fusion Tables) In Spring 2012 CartoDB upped their technology, API, UI, with [[leaflet.js|http://leafletjs.com/]] as the backend, also lets you do hover states, zoomable maps, more powerful computation than fusion tables.
[00:51:35.14] To do: examine MK Overview log file.
[00:51:35.09] MK Q: Are there updated docs for Overview desktop version?
For methodology and methods describing research context, data collection and analysis, refer to [[my project proposal|Document mining in data journalism]].
In brief, the purpose of this project is to collect information from ''data journalists'' using the ''Overview prototype'' in the course of their ongoing work w.r.t. ''document mining''. At the high level, we want to focus on the culture of data journalism and document mining in journalism, their workflows and processes, tool use, their technical backgrounds, domain expertise, collaborative activities, availability of data, exploration and orientation, time constraints. With regards to the use of Overview, we will focus on issues of utility/efficacy, usability, learnability, adoption into existing workflows and processes, comparison with previous processes.
My interview foci and associated questions below can be lumped into 2 broad categories: understanding the user as a data journalist and understanding their use of Overview. Some questions will come about naturally as follow-on questions, so the list is in no way a linear interview script. Our interviews will be subject to time constraints, so we should continue to iterate on these questions and determine which are the most important to ask, given possible variations in interview duration.
Questions are numbered. Bullet list questions are follow-up / elaboration questions and probes to be asked, if relevant.
Before the interview, the ''~Show-and-Tell'' video walkthrough session, where the user guides us through their analysis process. This can be facilitated with Skype Premium screen-sharing. For local participants, this can be done in person with basic screen capture recordings.
!Foci
!!On Overview
!!!Utility & efficacy
What [have / are] you [used / using] Overview to do?
>@@color:#bb0000; ''MB'': This might be obvious, but worth exploring possible answers to "what //could you have// used Overview to do?"@@
>@@color:#bb0000; This and following Qs to be asked during/after screen-sharing walkthrough@@
*Can Overview ingest some or all of your data? How long did this take? How much time did you spend cleaning/formatting the data specifically for Overview?
*How did you explore your data using Overview? How long did this take?
*How did you search your data using Overview? How long did this take?
*Did you have hunches about a story a priori? Were you able to verify these hunches?
>@@color:#444bbb; ''JS'': Good, but might need language tweaks. Reporters don't have hypotheses about the data, but hunches about the story. Otherwise they wouldn't have gone through the trouble of obtaining the data. However, I am speculating here. We should probably find more neutral questions that will evoke the user's own language @@
*Did you develop new hunches about your story while using Overview?
*How did you choose which documents to read? Depth vs. breadth? How long did you spend employing these strategies? How many individual documents did you skim / read?
>@@color:#444bbb; ''JS'': Is it fair to say that sampling strategy = "how did you choose which documents to read"? And of course we should collect usage logs here.@@
*''[Overview-specific question]'': How did you go about tagging your data? How long did this take? How many tags did you generate? How many did you delete? How many did you consolidate / split? How many of these tags were structural vs. semantic?
*''[Overview-specific question]'': What level(s) of tree pruning were used?
>@@color:#444bbb; ''JS'': Yeeess.... but I don't think the tree pruning works well as designed and hope to improve it. Which brings up an important point: the software is going to be a moving target, and needs to evolve between users. How do we account for that? @@
*Context of use: where were you while using Overview? What other applications did you have running? Were you taking handwritten or digital notes? Were you using Overview alone or in collaboration with others?
*How long did you use Overview for? How long was spent reading documents vs. organizing, browsing / sorting / tagging?
**Did this match your timeframe expectation prior to using Overview, given the size of the dataset?
**How much time was spent reading a single document? Did this vary?
*What proportion of documents did you read/skim? Are you confident in this proportion? Why or why not?
*What proportion of documents did you tag? Are you confident in this proportion? Why or why not?
*Not enough tags vs. too many tags?
*How did you deal with [unique / weird / important / unimportant / relevant / irrelevant] documents? (may be overlapping categories)
**How did you dismiss the unimportant? The irrelevant AND unimportant?
*Did you ever flag documents for later follow-up? Did this eventually happen?
*Did you ever re-read documents? Was this intentional?
*How, if at all, did you use the sampling functionality to read documents?
!!!Usability
Let's discuss Overview's usability issues.
*On installing Overview
*On preprocessing data / ingesting data
*Your first use of Overview: Orienting yourself within the UI
*''[Overview-specific question]'' Usability issues w.r.t:
**Linked displays, selecting items
**the tree display
**the MDS display
**UI for sampling
**UI for tagging
**UI for reading / opening in the browser
*What UI features did you like? Which didn't you like?
!!!Learnability
Related to usability, lets talk about learnability and Overview.
>@@color:#444bbb; ''JS'': Realistically, for at least the next few months and possibly longer "how did you learn" will probably be "Jonathan showed me." @@
*Learning materials: self-exploration vs. relying on JS's blog posts and instructions, other sources? How much time spent on each?
*Developing a conceptual understanding of item layouts in the MDS display and the tree display.
>@@color:#444bbb; ''JS'': Ah now this is a good one. Anecdotally, folks seem to have trouble interpreting these displays. For Overview it's an ongoing project to refine the language we use to describe what they're doing. @@
!!!Adoption
Has Overview become part of your workflow?
*[if not] do you foreseeing it becoming part of your workflow? Will it replace / add to / complement your workflow?
*How often is there a document set where Overview could be used?
>@@color:#444bbb; ''JS'': But also, could go the other way: someone might obtain a docset they may have otherwise ignored because they know they can now analyze it. @@
Would you recommend use of Overview for colleagues?
*Do you expect them to you use it in their workflows?
!!!Comparison: before and after Overview
!!!!Improvements
Has Overview improved your process? Do you see Overview as having the potential to improve your process?
!!!!Problems left unaddressed
What can't Overview do for you? What problem remains unaddressed?
!!!!Previously untouched data
Were you able to ingest/analyze data that you couldn't approach with other tools?
!!!!Previously unapproachable tasks
Were you able to perform tasks with Overview that you couldn't do (or couldn't do efficiently) with other tools?
!!!!Discoveries with Overview
Have you made discoveries (in your data) using Overview that you wouldn't have been able to make with other tools?
>@@color:#444bbb; ''JS'': Another way to phrase this issue is, what type of knowledge does Overview generate? My feeling is that it is something more/other than locating specific documents -- in the same sense as a "summary" is not a collection of good excerpts -- but this is hard to get a handle on.what type of knowledge does Overview generate? My feeling is that it is something more/other than locating specific documents -- in the same sense as a "summary" is not a collection of good excerpts -- but this is hard to get a handle on.@@
!!On data journalism and document mining
!!!Who: background & expertise
What brought you to working in this type of journalism?
>@@color:#444bbb; ''JS'': Yep, important stuff. This sort of demography of data journalists has never been done to my knowledge, so independently useful. However it does raise the question of sampling/recruitment. If we stick strictly to folks who attempted to use Overview for something, obviously that does skew it in some way, if only because of the specific data type that Overview is built for (unstructured, text, "documents.")@@
!!!!Domain expertise
*Relevant demographic information: age, number of years working in journalism, number of years working in data journalism, education background
!!!!Technical background
*Re: data journalism: self-educated vs. formally educated, trained, or mentored, mixed (discerning between what was formally taught and what was independently learned)
*Technical skill set: spreadsheets and table manipulation, data cleansing, internal and external validation; programming / scripting experience
**skill set w.r.t. document mining and unstructured data vs. skill set w.r.t. structured data
!!!Context of data journalism
Tell me about your current day-to-day work.
!!!!Context
*''Career context'': agency / bureau affiliation, past and present
*''Spatial context'': where does the work happen? (office setting, working remotely, other)
*''Task context'': what else is going on? multi-tasking vs. single-task focus?
*''Temporal context'': project timeframe + deadlines: hours, days, weeks, months, vs. ongoing investigation
!!!!Collaboration
*the extent to which work is collaborative
**w.r.t. data acquisition, data pre-processing, analysis, writing
**reliance on technical expertise of colleagues
!!!Workflows and processes
I'd like to hear about a recent story of yours, one from before using Overview, involving large collections of documents, representative of this type of work.
*(If there is no precedent, can I hear about a story involving large amounts of data analysis?)
*Is this example representative or unique?
*(please provide links to articles, or send manuscripts)
!!!!Methodology
*data collection and analysis methodology prior to Overview: personal vs. adopted / taught? strictly followed vs. ad hoc variations, subject to constraints of data, deadlines?
>@@color:#444bbb; ''JS'': This is tricky. First, I'm going to be teaching them how to use the tool, so in that sense it's our methodology, not theirs. But also the whole issue of "where does a story come from?" I feel that several discussions of grounded theory touch on similar points.@@
>@@color:#bb0000; ''MB'': this question refers to one's existing methodology, their use of current tools.@@
!!!!Tool use
*Tools/services for data collection (e.g. ~DocumentCloud)
**How long does it take to collect data, using these tools/services?
*Tools for pre-processing: scraping / separating ~PDFs or semi-structured text, cleansing, pre-processing, internal and external validation (e.g. Google Refine, Google Fusion tables, spreadsheet manipulation)
**How long do these processes take, using these tools?
*Analysis tools (e.g. tools for keyword search, data exploration/orientation, visualization?). For each tool:
**What did you use the tool to do?
**Were you able to ingest some or all of your data? How long did this take? How much time did you spend cleaning/formatting the data specifically for this tool?
**How did you explore your data using this tool? How long did this take?
**How did you search your data using this tool? How long did this take?
**Did you have hunches about a story a priori? Were you able to verify these hunches?
>@@color:#444bbb; ''JS'': Good, but might need language tweaks. Reporters don't have hypotheses about the data, but hunches about the story. Otherwise they wouldn't have gone through the trouble of obtaining the data. However, I am speculating here. We should probably find more neutral questions that will evoke the user's own language @@
**Did you develop new hunches about your story while using this tool?
**How did you choose which documents to read? Depth vs. breadth? How long did you spend employing these strategies? How many individual documents did you skim / read?
**''Context of use'': where were you while using this tool? What other applications did you have running? Were you taking handwritten or digital notes? Were you using this tool alone or in collaboration with others?
**How long did you use this tool for?
**''Usability issues''
***On installing this tool
***On preprocessing data / ingesting data
***Your first use of this tool: Orienting yourself within the UI
***What UI features did you like? Which didn't you like?
!!!!Data
*When collecting the data, what state was it in?
**Individual documents? Single document?
*When analyzing the data, what state is it in?
**Unstructured text vs. mixed / semi-structured vs. structured? For the latter two, categorical variables, numerical variables, mix?
>@@color:#444bbb; ''JS'': So this suggests to me you ARE interested in "structured" data. Of course we have to define that too. And some datasets are mixed, e.g. war logs which is a table with 17 columns, several of which are free text while others are categorical or numeric.@@
**Are individual points (documents) tagged or untagged? [for structured data] do categorical variables exist to tag each point?
**How do you find correlations between structured and unstructured information?
>@@color:#444bbb; ''JS'': Yeah, good thing to track. We (and leaksplorer) use categorical variables from one column to color the points placed by text analysis. This is a great way of finding correlations between structured and unstructured information.@@
!!!!!Provenance
*Where did the data come from? Who manufactured it? Who collected it or consolidated it?
**Who did the cleansing / preprocessing [if not yourself]?
*[if tags] Where did the tags come from?
!!!!Orientation
*re: Orientation, I mean getting an initial overview of a dataset of unstructured text documents, how do you orient yourself within a dataset? Does orientation matter?
*How do you get a sense of the high-level structure of a dataset? The size of a dataset?
>@@color:#444bbb; ''JS'': Good, though if we're going to try to probe for techniques we suspect in common use, I'd add word clouds, and word frequency visualizations generally. Also mapping, when the data has a geographical component.@@
**To what extent do you rely on metadata? tags? word counts? word co-occurrences? word clouds? word frequency? mapping (when points have a geographical component)
**Is visualization used?
!!!!Exploration vs. targeted search
*Relative time spent exploring vs. targeted search
>@@color:#444bbb; ''JS'': For journalists, I am not convinced that there is a clear conceptualization of hypothesis validation. Language is certainly wrong. "Verification"? "Fact checking"?@@
*How much of your work is hypothesis generation / exploration and how much is verification / fact checking?
**What other sources were used to complete the work?
***other texts/data, vs. communications/interviews with people.
***When were other sources used? For hypothesis generation? For fact-checking/verification/corroboration? What was the pipeline/workflow for using other sources?
>@@color:#444bbb; ''JS'': One important missed point: what OTHER sources did you use to complete the work? I suspect that documents are very rarely used on their own. For that matter I might argue that a story based only on documents was bad journalism, for the same reason you don't rely on single sources for important conclusions. Other sources might be broken down into two categories: other texts/data, versus communications/interviews with people. This step is important for data validation too.@@
!!!Results
What constitutes a strong piece of data journalism involving document mining?
>@@color:#444bbb; ''JS'': An ongoing conversation. Very broad. Again, what is "data journalism"?@@
!!!!Reader perspective vs. peer perspective
>@@color:#444bbb; ''JS'': Interesting!@@
*How should readers react to document mining in journalism? What do the reader comment boards say?
*How should document mining in data journalism, its goals and processes, be reviewed and critiqued be peers? How has your data journalism been reviewed / critiqued?
!!Framing: the ~MoDisco ~EuroVis reviews
R4 says: evaluation needed, a head-to-head comparison against other tools, for example:
*Oesterling et al (2010) (topic islands),
*Yang et al (2010) (Newdle)
*Chen et al (2009): (Examplar-based Visualization)
User studies are also needed (usability, utility/efficacy, etc.)
>//do users qualitatively prefer ~MoDiscoTag to a state of the art system, and if so, how/why?//
>[...]//The case-studies are interesting, but their contribution would have been better supported by a more in-depth comparison with how existing techniques would have foundered in these situations where the proposed methods allowed further useful analysis//
*@@color:#444bbb;users are not using these tools; they are using keyword search; many using [[Document Cloud|https://www.documentcloud.org/home]]: includes a searchUI, allows for collaboration, annotation, tagging articles and embedding frames, private and public sharing (example: [[ProRepublica's|http://www.propublica.org/]] use of embedded frames for source tracking back to original documents from [[Document Cloud|https://www.documentcloud.org/home]])@@
!!Key Questions
//who and where? / who are the users? where are they?//
*is JS a gatekeeper or representative user? @@color:#444bbb;
*not a representative user; middle-ground/ on-the-fence between designer and user
@@
**is JS a DR math expert or middle-ground user? somewhere in between?
*who are the other users?
**other AP staffers?
**distributed hobbyists?
**somewhere in between?
**programmers / journalists / DIY vis people?
**DR math experts or middle-ground users? somewhere in between? a mix?@@color:#444bbb;
*primary target users are journalists, little programming experience, some learning to program, however just as many who aren't
*overall, not tech savvy (e.g. struggling with installation, unsure how to scrape a PDF, a website)
*stereotypical user: newsroom background, 40ish, standard career path, at some point in last 10 years faced with big data and realizes "ooh, data"
*core skill set:
**high familiarity working with spreadsheets, Google Fusion table (handles big data), Google Refine (data wrangling, faceting, cleanup, splitting and merging columns) - only works up until ~7k rows
**knowing what to ask for, how to ask for it, who to ask
**data cleanup
**internal validation - empty fields, outliers
**external validation - examines how source data was generated, inter-rate reliability, incentives to manipulate data
*secondary target users are hacker-types, trying to establish a developer community
**wants to use this open-source community as a hiring pool
*Overview shown internally to AP bureau (internal show and tell) in NYC;
**interested in methods and techniques, but [[story produced|http://overview.ap.org/blog/2012/02/private-security-contractors-in-iraq-analysis/]] wasn't very compelling, just confirmed anecdotal stories and congressional hearings which have been widely publicized
**a few users at AP may be using the tool soon, big data released all the time, will become necessary to use such tools
**representative of a cultural shift towards understanding comprehensively, rather than anecdotally
@@
//why? / what is the purpose of Overview?//
*data analysis
*hypothesis generation vs. validation@@color:#444bbb;
*both
*new FOIA laws require information to be searchable (the government must have a user interface), however this requires you have to know what you're looking for a priori; bad if you want to explore or do both simultaneously
*what is needed is a search UI and an exploration UI
*a web app, publicly accessible, no installation required
*source tracking and embeddable frames within a news article
*orientation and sense making
@@
//what is the current practice?//
*what is he doing?
*what are the users doing?
*existing workflows?
**Overview to replace vs. add vs. complement?
*what's the deal with tagging?@@color:#444bbb;
*5 people currently installed, no feedback yet
*no one's using Jigsaw (too slow)
*adds the ability to explore to current workflows which are mostly searching for known topics;
*orientation, sense making, exploration missing from large document sets
*what tech-savvy journalists need derived from informal discussion (i.e. Sarah Cohen - we need computer-assisted clustering of topics without knowing a priori what to look for - see Grimmer + King article: simple techniques point to structure, help journalists orient and make sense of a data set)
*his current practice, used for the Iraq contractors research:
**tagging of documents (~12 hours)
**random sampling (~8hrs)
**lucky to read 1% of the data, Overview provides a method for deciding what not to read
@@
//what do we know about the data?//
*provenance
*assumptions
*cleansing
*assumption of unrevealed structure @@color:#444bbb;
*most data not hidim, but new hidim data released all the time
*OCR data, incomplete, garbled, can't process images, handwritten notes
*comments on news message boards (i.e. ~HufPost), not hidim, no interesting structure in Overview
@@
//how are current processes performed?//
*~MoDisco, others?
*has he tried the techniques mentioned by the ~MoDisco reviewers?@@color:#444bbb;
*No. existing workflows don't include or facilitate exploration of a data set, orientation
@@
!!Discussion points
//long-term goals//
*how does JS document success? what is the realization of JS's vision?@@color:#444bbb;
*are journalists able to publish stories with the tool?
*not anecdotal, but comprehensive stories
*quantitative and qualitative - if 10 people in 10 different newsrooms are able to use it, he's happy
@@
*goals for collaboration
//why// we want to be involved
*1.5+ full papers' worth of material, perhaps more?
**~MoDisco/Overview design study validation
**Overview deployment study
**my ~PhD interests:
***evaluation for ~InfoVis
***graphical perception: human pattern perception and apophenia
***divided attention (multitasking and interruptions)
***learning and expertise
***insight and problem solving
**scope / what we can address: utility, usability, learnability, adoption
//what// we are proposing to do
*mixed method evaluation
*in situ and remote
*qualitative and quantitative methods:
**participant observation
**interviews (in-person, phone, Skype),
**contextual inquiry
**focus groups
**artifact analysis
**think aloud protocol
**benchmark task set user study
**[[insight-based evaluation|Information Visualization Evaluation: Qualitative Methods]] (comparison against other tools?)
*a field study validation of Overview as validation for design study paper, which could inform a larger quantitative insight-based evaluation@@color:#444bbb;
*head-to-head comparison with keyword search
**question of inter-rater reliability
**will two journalists given the same source material produce the same story?
**implies an objective goal - how to prove?
@@
//deployment timeline//
*funding timeline
*our timeline / RPE timeline
*is there a fit?@@color:#444bbb;
*he has means to continue development, UI refinement, generating a web-accessible interface
*do users exist that can work with us?
@@
!!Next Steps
*field data collection - continue to solicit feedback from journalists downloading the tool, rationalize the design of overview post-hoc (impossible to realistically conduct pre-design requirements analysis at this stage, especially when tool adds to a task flow, not something that replaces an old one)
**to do: generate a list of questions re: use case/user story (scenario), usability, barriers to adoption,
***characterize what problem they're trying to solve, what is the data, pre-processing needs to be done, how long of a process
***what have they attempted to do to solve the problem that they couldn't have attempted before using the tool
***background information: previous experience in data journalism, level of technical expertise, what tools they've used in the past
**collect usage data, their stories, their tags, their clusters, other metrics
*in parallel, develop an insight-based methodology to evaluate the tool
**within-subjects comparison of keyword search vs. keyword search + overview - what do they find? what new discoveries does the tool provide?
**compare notes (quantitative) and final stories (qualitative), the latter the eventual goal of the tool - can it be used to write compelling stories?
**recruit journalism / data journalism students (i.e. Columbia, Hong Kong, ~McGill all have data journalism schools)
***self-selection bias
**to do: begin considering methodology
***piloting over the summer, establish rapport with data journalism schools, introduce into curriculum in the fall
''Note'': unofficial off-the-record phone conversation re: WIP stories; identifying details have been removed. No recording (voice / screen sharing) was made. To follow-up after stories run, clearance from editor/manager to participate and share logs, record audio and screen, share document sets.
!!Previous completed project (Jul '13)
*''the document collection'': 4,000 pages of email from FOIA request between gov't agencies, lots of duplicate emails (cc, reply-all, fwds), attachments
**''format'': single 4,000 page PDF downloaded from link supplied by gov't agency
**''pre-processing'': uploaded single PDF to ~DocumentCloud, OCR didn't process attachments (images of spreadsheets, expense claims/receipts, turned sideways in PDF); imported into Overview with 'single page = single document' import option.
*''the goal / hypothesis / hunch'': finding the smoking gun emails
*''the approach'': confirming what had already been found during exhaustive read of all documents; person of interest appeared at 3rd level in topic tree; user found a topic that they were unaware that they should be looking for
**''tagging'': unable to see tag bar in [[Palemoon|http://www.palemoon.org/]] browser, tags didn't show up in tree (resolved by using Overview in Chrome)
**''reading'': prior to import into Overview, user read all 4,000 pages in about 2 days, able to skip many duplicates (cc / reply-all), junk attachments
**''duration of use'': used Overview for a short time in July
*''the outcome'': story ran at the end of July; not well recieved by gov't agencies that it criticizes
!!WIP Overview work (Aug '13, follow-up to Jul '13 story)
*''The document collection'': 4 different document collections from 4 different FOIA requests, one Overview project for each
**''pre-processing'': as before, impoorted as 4 separate ~DocumentCloud projects
**''format'': arrived on 4 ~CDs, up to 70 multi-email ~PDFs on each, each roughly 2,000 pages of email
*''the goal / hypothesis / hunch'': to follow-up on earlier story, to see what else is in each document collection, to find key actors and decisions made, as well as the context for these decisions; to "find several smoking gun" emails; unable to use keyword search since user doesn't know what the terms would be a priori (in retrospect, user suspects a possible keyword that might lead to the same set of smoking gun documents); gov't agencies not forthcoming about decisions and key actors / events
*''the approach'': (in progress) read everything, tag everything, but skip the obvious junk (badly OCR'ed attachments, non-email content (e.g. expense report spreadsheets), skip the duplicates); drill down to smallest cluster size; once entity / person of interest is found, does a keyword search for that person in ~DocumentCloud (was unaware of Overview's new search feature); Q: did oyu find anything you weren't looking for? A: yes, found a topic that had previously been unaware of
**''tagging'': tag everything; currently has 2 content-based tags and a "skip" tag, the latter indicating emails/attachments not worth reading, compulsive about tagging everything
**''reading'': reading all the non-junk/non-attachment/non-duplicate content; admits to being compulsive about this
**''duration of use'': no deadline, has been working steadily on these 4 collections since Monday/Tuesday Aug 19-20, up until and during our conversation
*''the outcome'': TBD (will be in touch after story runs) "haven't found the golden nugget yet"
!!General comments
*thinks Overview is great
*''usability'':
**couldn't see colour tags / tag bar in [[Palemoon|http://www.palemoon.org/]] browser, tags didn't show up in tree (resolved by using Overview in Chrome); document viewer would go blank, contents would return on browser refresh: this too resolved by switching to Chrome.
**wasn't aware of "back to list" option in document viewer, though this is shown upon initial opening of a document set, this is an easy way to see the keywords for an entire cluster
**wasn't sure where the tags went upon closing/opening a document collection (tags need to be selected to be seen in the tree, only a single tag is shown at a time in the tree, though documents having multiple tags have coloured labels in document list/viewer)
**wasn't aware of new full-text search and search-results-to-tag features; amazed by this when discovered
**realized that if you re-import a document set to get new full-text search feature, you lose your tags
**why does cluster hierarchy only go 10 levels deep? found a node at level 10 w/ 178 seemingly unrelated docs - shouldn't these break apart more?
**''conceptual understanding of how Overview works / trust in clustering algorithm'':
***slowly building trust in Overview's ability to sort documents by keyword; wasn't sure initially, but SOME/ALL/MOST keyword labels helped convince user of Overview's ability to cluster correctly, reading through a node with the MOST/ALL label cemented this trust
***didn't have an understanding of underlying NLP: wondered if Overview just throws out article words, retaining nouns?
*previous stories where Overview might have been useful
**yes, FOIA document collections and email corpuses common now; in the past, uploaded these to Google Docs (faster upload than ~DocumentCloud, but poor OCR, meaning ineffective keyword search); used Google Docs collaboratively, ability to read documents immediately, especially for stories with deadlines (waiting overnight for a ~DocumentCloud import is too long, even for a 2 week story, admits this is just impatience, likeining this to a microwave and the "I want it now" mentality)
*discovered Overview in [[program notes|http://ire.org/events-and-training/event/21/964/]] of [[Investigative Reporters and Editors (IRE) conference|http://ire.org/conferences/ire-2013/sessions/]] (Jun '13), notices immediately its relevance to current work; watched an Overview demo video (an outdated one, not the July '13 walkthrough video), didn't read Overview blog posts: wasn't aware of them / previous stories done with Overview
*reagarding stories with deadlines, 2 weeks is the norm (not 2 days), often when competing with other newsrooms having the same FOIA data: a race to publish, competitors have to read the same documents that you have; other times, when other newsrooms don't have the same documents, no deadline is given; editor is excited and curious about using Overview so time can be spent using it.
*user background: not familiar with CAR, more of a traditional newsroom background
**other tools used: ~DocumentCloud, Google Docs / Drive
Source: <<cite Becker1957 bibliography:Bibliography>>: realizing the limitations of interviews, the benefits of participant observation, and contextual interviews.
Two of the methods we use in HCI are the [[Think Aloud Protocol]] and [[Contextual Inquiry]], the former akin to observation while the participant describes their activities in real time, and the latter allowing the interviewer to probe and ask questions as a participant carries out an activity.
*Learning the native language
It is often true that while abstract tasks and artifacts are the same across many data analysis domains, domain-specific language may be used instead. This is definitely the case in the [[HDD-DR Ethnographic Project]], in which bioinformaticians have their own set of terminology for statistical techniques and analysis methods, which are in actuality identical or very similar to techniques described in other domains.
*Matters Interviewees are unable to talk about
This is likely to come up in situations where multiple stakeholders are involved, particularly during the requirements analysis phase of a project. For instance, in the [[Vismon]] project, we had hoped to interview / observe managers and policy makers (the target users) and their subordinate scientists (the tool designer), which could have made for some interesting dynamics between superiors and subordinates.
*Things people see through distorting lenses
In longitudinal HCI studies, we often study individuals and contexts before, during, and after an intervention is introduced, as in much of [[Action Research]]. Tracking change and discrepancies in this timeline should be observed, and we should not solely rely on interviews and diary studies/usage logs.
*Inference, process, context
A difficulty I had during the [[HDD-DR Ethnographic Project]] is that I wasn't present for the original interviews and contextual inquiries, and was left with partial transcripts and observer/interviewer notes. Thus the inferences I had to make and the uncertainties re: terms and practices was even larger than the interviewer.
!!References
<<bibliography>>
!!!Conceptual Understanding
''Principal Component Analysis'' finds dimensions which are linear combinations of original dimensions that capture the most variability in the data (rotation of the axes along the axes of maximal variation), wherein the first dimension captures the most variability, while the second and subsequent dimensions (each being orthogonal to the preceding dimensions), capture the most of the remaining variability in a diminishing fashion. After this transformation the variables are considered to be independent.
A [[Scree Plot]], a histogram of dimension variability depicts each eigenvalue as a sequence from largest to smallest, will indicate the number of dimensions contributing the most variability. An observer can decide a threshold which accounts for most of the variability, thereby discarding remaining dimensions until only //k// variables remain. This can reduce noise in the data. However the effect of outliers should signal a note of caution.
There is no obvious correspondence between components derived from correlation and covariance matrices.
''Nonlinear PCA'': PCA can also be used for identifying ''principal curves'', where is its suspected that relationships between variables are non-linear.
It is often confused with [[Factor Analysis]] - the underlying math is quite dissimilar, although the techniques produce similar results. PCA does not have an underlying statistical model. Poorly correlated data will produce meaningless results with FA, while it will produce components similar to original variables with PCA. PCA is also a bad idea when there are fewer, more distinct variables.
!!!Terminology I don't understand
*eigenanalysis
*sphering
*Kronecker's delta function
*shapelet functions
*Hotelling T^^2^^ test
*rotation procedures:
**varimax, quartimax, equimax
**oblique: oblimin, quartimin, covarimin, HK, promax, maxplane
**Procrustes (particularly useful in comparing alternative MDS solutions
!!!Sources:
<<cite Holbrey2006 bibliography:Bibliography>>
<<cite Munzner2011>>
<<cite Tan2006>>
This model accounts for how long a casual visualization is used and whether use is repeated, classifying uses as approach (unrecognized potential use), recognition (acknowledged potential use but not used), single-use short-duration, single-use long-duration, repeat-use short-duration, and repeat-use long-duration. Where a visualization sits among these usage instances depends on promoters (motivating forces) and inhibitors (other forces).
Source: <<cite Sprague2012 bibliography:Bibliography>>
Observation and detailed coding of a protocol (i.e. a series of tasks and subtasks) based on pre-defined codes based on stages of the protocol. Data is collected from observational notes from domain experts, video recordings of interactions performed, audio recordings of utterances, collection of written and typed deliverable artifacts.
<<list filter [tag[pulse]]>>
!Energy Manager Home
!!Overview
*//what//:
**date ranges, spaces (buildings or building groups, hierarchical; with square footage)
**resource types (electricity, steam, hot water, water, natural gas, others)
***water treated specially (non-energy resource measured in volume only, not included in combined consumption)
**raw values (consumption, demand, outdoor temperature)
**derived values: energy savings, consumption per area, energy waste, heating/cooling degree days (temperature measurements relative to average baseline temperature), performance rankings relative to baseline (poor, normal, great)
**unit selection (KWh/GJ, m/ft, 銪//how// (Overview)
**''encode'': n/a
**''select'': building (group), date range
**''navigate'': n/a
**''arrange'': n/a
**''change'': unit selection (propagated to charts below)
**''filter'': by resource type, by date, by building (group)
**''aggregate'': by date range, by building group (segregation not supported); aggregation propagated to charts (below)
**''annotate'': n/a
**''import'': n/a
**''derive'': n/a
**''record'': n/a
*//how// (//Combined Consumption//):
**''encode'': stacked bar chart (consumption, colour mapped to resource type) + line chart (temperature), date along x-axis)
**''select'': hover details-on-demand for each time-series interval in visualization
**''navigate'': semantic drag zoom, zoom out button
**''arrange'': n/a
**''change'': n/a
**''filter'': by resource type, by date
**''aggregate'': by resource, other aggregation propagated from Navigation Bar (by date range, by building group (segregation not supported); aggregations also propagated to Energy Savings
**''annotate'': n/a
**''import'': n/a
**''derive'': energy consumption consumption by resource -> combined energy consumption
**''record'': export chart options (PDF, PNG, JPEG)
*//how// (//Energy Savings//):
**''encode'': dual positive/negative bar chart (energy savings vs. energy waste) + time series scatterplot (HDD/CDD), date along x-axis
**''select'': hover details-on-demand for each time-series interval in visualization
**''navigate'': semantic drag zoom, zoom out button
**''arrange'': n/a
***''change'': n/a
**''filter'': by derived value, by date
**''aggregate'': by date range; other aggregation propagated from Navigation Bar and Combined Consumption (segregation not supported)
**''annotate'': n/a
**''import'': n/a
**''derive'': combined energy consumption -> HDD/CDD, energy waste, energy savings
**''record'': export chart options (PDF, PNG, JPEG)
*//how// (//Performance Breakdown + Ranking//):
**''encode'': dual positive/negative bar chart (energy savings vs. energy waste) + time series scatterplot (HDD/CDD), date along x-axis
**''select'': select row/column/cell in //Performance Breakdown// to ''filter''; select row in //Performance Ranking// to select building or building group (filter out other buildings)
**''navigate'': n/a
**''arrange'': sort performance ranking table by % savings, building (group) name, resource type, savings, consumption, consumpion per area
***''change'': n/a
**''filter'': by //Performance Breakdown// row (resource type), by column (all resources - poor / normal / great), by cell (resource type + ranking); filtering propagated from Navigation Bar
**''aggregate'': aggregation propagated from Navigation Bar (segregation not supported)
**''annotate'': n/a
**''import'': n/a
**''derive'': n/a
**''record'': export //Performance Ranking// table (CSV), no export for //Performance Breakdown//
!!Demand
*//what//: resource type (electricity, steam, hot water, water), temperature, time period, building (group), unit
**derived values: baseload, peak demand, average demand, peak demand value for time interval, min temperature for time interval, max temperature for time interval, avg. baseload for time interval
*//how// (Demand):
**''encode'': colour-coded time-series line chart
**''select'': hover details-on-demand for each time-series interval in visualization
**''navigate'': semantic drag zoom, zoom out button
**''arrange'': n/a
***''change'': n/a
**''filter'': hide/show derived values and temperature, filter by date; other filtering propagated from Navigation Bar (building (group), time interval)
**''aggregate'': aggregation propagated from Navigation Bar and Combined Consumption (segregation not supported)
**''annotate'': n/a
**''import'': n/a
**''derive'': demand -> peak demand, average demand
**''record'': export chart options (PDF, PNG, JPEG)
*//how// (//Performance Breakdown + Ranking//): (as above)
**rank by % savings, name, resource, avg. baseload, avg. demand, peak demand
!!Consumption
*//what//: resource type (electricity, steam, hot water, water), temperature, time period, building (group), unit, cost, actual consumption, forecasted consumption (for future dates)
*//how// (Consumption):
**''encode'': time-series bar chart with cost and temperature line overlays, forecasted consumption values have lower alpha value
**''select'': hover details-on-demand for each time-series interval in visualization
**''navigate'': semantic drag zoom, zoom out button
**''arrange'': n/a
***''change'': n/a
**''filter'': hide/show values, cost temperature, filter by date; other filtering propagated from Navigation Bar (building (group), time interval)
**''aggregate'': aggregation propagated from Navigation Bar and Combined
Consumption (segregation not supported)
**''annotate'': n/a
**''import'': n/a
**''derive'': forecasted consumption
**''record'': export chart options (PDF, PNG, JPEG)!!Benchmarking
*//how// (//Performance Breakdown + Ranking//): (as above)
**rank by % savings, name, resource, savings, consumption, consumption per area
!!Benchmarking
*//what//: energy consumption of currently selected space (broken down by resource), computed as an Energy Performance Index (EPI), //calculated by dividing a space's energy consumption by floor area. An EPI ranking based on database of similar spaces of the same use type, size, and geographic location//.
*//how// (Ranking, static visualization:
**''encode'': percentage rankings are represented along a 1D horizontal glyph in which the top 25% EPI rankings (left) are coloured green (GOOD) and the lower 25% EPI rankings (BAD) are coloured red. A black triangle superimposed on this glyph indicates the percentage ranking of the currently selected space and resource type.
**''select'': space (propagated from main navigation menu), dates (annual values propagated from date selection):
>//The benchmarking values are calculated using the last 12 months* of data from the end date in the date bar. Therefore if you are using any of the quick select date options (yesterday, last 7 days, last 3 months, or last 12 months), the exact same data is being used. You can however, manually select the data used by changing the end date on the date bar.//
Benchmarking data is available for electicity, but not for steeam or hot water. At the campus level, it asks you to configure a floor space, taking the user to the Configuration page (below)
!!Alerts
*//what//: reports of consumption values exceeding thresholds within the selected date range (organized by space)
*//how//: drill down ''selection'' of a space from a hierarchical menu of spaces (either in the main navigation or in the main Alerts panel). Multiple alerts for the same space and date range include charts for each threshold alert, which are located in the "Thresholds" section of the Configuration page (see below)
!Engagement Dashboard
!Management
!Reporting
!Configuration
*Configuration of floor spaces for benchmarking
*Specification of alert thresholds and threshold charts
!Sources
*[[Pulse Energy Manager|https://ca.pulseenergy.com]]
*[[help guide|https://ca.pulseenergy.com/help/home/home]]
BE is the Head Maintenance Engineer (Automation) for UBC Building Operations. He is a veteran of building operations and jokes that he's been around since the beginning (pointing to a 1928 photograph of UBC under construction on the wall by his desk, in which the original steam lines are being dug). He's witnessed a change in mandate re: building operations over the past several years, from a mentality of "keeping everyone comfortable" to one of "minimizing energy costs". It used to be that all buildings were running an full capacity at all times, and if one occupant complained that a building was too hot/cold, Building Ops would address the situation in order to make the occupant happy; now, this isn't done. UBC building operations is in flux with the Continuous Optimization program, the ongoing conversion from steam to hot water (District Energy hot warer energy project), the hiring of an energy analyst and subsequently changing his job description, disagreements between consultants and full-time building management staff (e.g. teh recent Buchanan Tower refit).
Much of BE's time is spent responding to work requests and complaints from building occupants; this process was formerly all done using paper work request forms. Recently, this process has transitioned to a paperless reporting system through UBC SSC, which makes requests and responses easier to track and comment on. When a request is made, BE will review the building in their Building Management System (BMS) reporting software, and in Pulse Energy Manager.
The BMS reporting and management software they use is [[Siemens' Apogee|http://w3.usa.siemens.com/buildingtechnologies/us/en/building-automation-and-energy-management/apogee/software-and-web-applications/insight-advanced-workstation/Pages/insight-workstation.aspx]], which breaks down a campus map into buildings by BMS type, and breaks down a building into its components via floor plans and steam flow charts; floor plan sections are colour-coded, corresponding to different HVAC systems serving them. The BMS will describe, for each building, sources of energy consumption. Not all of the (100+) buildings on campus are integrated with the Siemens BMS reporting system, as not all buildings are outfitted with a Siemens BMS (others are Johnson, Delta, Honeywell).
BE receives a daily email digest from Pulse, as well as a weekly trend performance report. The daily digest ranks buildings according based on the previous day's energy use and differential from the previous day, ranked from largest increase to largest decrease. These emails seem like a nuisance, especially since BE also receives alert threshold emails. When vigilantly examining several days' worth of digest emails, BE might notice a building exhibiting a high spike in consumption on one day when it hadn't previously demonstrated any such erratic consumption. This anomaly is sometimes indicative of a BMS failure or a energy consumption source left on when it ought to have been switched off. Often these anomalies can be blamed on power outages. When an anomaly cannot be explained, it is difficult to determine the cause of the anomaly without being intimately familiar with the particular building and its energy consumption sources (for a campus with 100+ buildings, it is really difficult to have expert knowledge of all buildings, particularly buildings with experimental equipment). If the anomaly is one in which the consumption curve differs from the baseline curve, BE requires an understanding of what time window the baseline curve was trained on, and whether any change to a building's BMS in the interim might have caused the difference. Some campus buildings are at various stages of the Continuous Optimization refitting program, and thus their baseline curves might be obsolete. In these cases, BE will contact JC @ Pulse and request that the baseline be recalculated using a more recent time window, omitting known anomalies (such as power outages or statutory holidays). BE does not create Pulse management charts nor does he configure buildings or baselines (Pulse's JC handles this currently).
BE currently uses Pulse passively, aside from setting up threshold alerts and email digests, he may also leave user notes on existing management charts. Email alerts will also direct BE to Pulse's management charts; alerts are mostly the result of overly-sensitive alert configurations, and many are not investigated; often the duration of the threshold event is shorter than the time it takes to investigate an event.
Pulse Energy Manager is "of marginal use" for BE, largely due to its failure to integrate with Siemens' BMS reporting software. A bidirectional link between Pulse and the Siemens BMS software would be a huge boon, one in which the energy consumption data could be understood in the context of a building's BMS structure, and vice versa, making it easier to allocate the source of energy consumption within a building. Like Pulse, the Siemens BMS reporting software also factors in weather variables and forecasting from Environment Canada. He suspects that Pulse Energy Manager would be better if there was a division of labour across all campus buildings, allowing users to specialize and become deeply familiar with a smaller set of buildings. Currently, the information in Pulse is of little use without this detailed knowledge, only some of which can be gleaned from the BMS reporting tool. You could spend all day working in Pulse trying to understand a single building. In practice, only 3-4 hours a week are spent using Pulse, spanning about 10 buildings a week, buildings involved in work requests and/or as flagged in emails/alerts. Of these, 3-4 specific buildings are routinely monitored each week (e.g. CEME, Brimacombe). Outdoor air temperature is infrequently considered. ~CUSUMs are occasionally viewed for longer-term consumption trends, though the longest time window considered tends to be about a week. He only uses single-building charts in the Management tab, since the multi-building charts and the Home tab (and its ranking) hide anomalies and make energy consumption performance seem better than it is (loss of detail).
BE also sees Pulse as being not "user-friendly" compared to other systems (see [[Pulse-LZ-13.07.09]] re: Schneider's [[PowerLogic|http://www.schneider-electric.com/products/ww/en/5100-software/5115-energy-management-software/2251-powerlogic-ion-eem-395/?APPLICATION=1024]] EMIS, [[Energent|http://www.energent.com/]], NRCAN's [[RETScreen|http://www.retscreen.net/ang/home.php]]). BE says that Pulse might be better suited for smaller portfolios. However, Pulse is mandated by BC Hydro, so UBC went with it.
Follow up with other building management staff: Paul Salikin, Justin Chia, Josh Waugh (hired as energy analyst).
meeting w/ [[JC|http://www.linkedin.com/pub/jerome-conraud/26/853/214]], energy manager @ [[McGill|http://www.mcgill.ca/facilities/utilities/dashboard]] via Skype + screencast recording
*[[McGill Energy Dashboard|https://my.pulseenergy.com/mcgill/dashboard#/overview]] for ~65 buildings
!!JC Interview Notes 13.07.29
JC and another energy manager are responsible for the analysis, reporting , and billing of ~McGill's two campuses. JC's colleague is primarily a liaison with building operations (service requests), while JC's role is that of planning, analysis, and reporting. Bi-monthly meetings are held with building operations to discuss issues that may have come up over the interim period, issues to which building operations can address.
JC's day-to-day operations include monitoring daily and hourly consumption patterns, beginning at the level of zones (four of which compose the main campus, with approximately a dozen buildings per zone). Like UBC, some zones are more erratic and unpredictable than others, particularly buildings with scientific or medial equipment; life sciences and physical sciences buildings. If an anomaly is spotted at this level, JC would enter the edit mode for the point data corresponding to the zone to determine which individual buildings compose the point. This is in contrast to an earlier approach in which a single flat list of buildings for the entire campus was navigated in an attempt to locate the building causing the anomalous consumption signature. There remains to exist a straightforward way to navigate between aggregate points and their component building points in the Management tab, though Pulse's new home tab will simplify this to some extent, with its hierarchical list of building groups. This problem is also encountered for the current home tab, in which clicking on a bar in the //Combined Consumption// stacked bar chart for an aggregate group of buildings does not reveal each component building's contribution to that bar.
The //Energy Savings + Waste// chart in the portfolio tab is seldom used because the values used to calculate energy savings and waste are not transparent, so they are difficult to interpret and believe.
Management charts with combined consumption and cumulative savings are not useful or particularly informative, but toggling back and forth between individual charts containing this information is tedious.
There is need for a greater understanding of the role of weather variables other than outside air temperature, most significant of which being relative humidity, and, to a lesser extent, wind speed, which is a factor in the winter months. At a higher level, it is assumed that these variables are taken into account when constructing baselines, however a greater transparency is needed to better understand how weather variables factor into these baselines, and which weather variables are important for particular points (buildings/resources). Like at UBC, the analytics that generate the baselines are too much of a black box.
Chart creation could be a simpler drag-and-drop experience, as opposed to the cumbersome navigation-heavy point and baseline configuration now required. Pulse's new design should alleviate this problem for most commonly-viewed charts.
Overall, Pulse Energy Manager is seen as intuitive and more approachable than some competitors' products, including building automation products by [[Rockwell|http://ca.rockwellautomation.com/]], [[Seimens|http://w3.usa.siemens.com/buildingtechnologies/us/en/building-automation-and-energy-management/apogee/software-and-web-applications/insight-advanced-workstation/Pages/insight-workstation.aspx]], [[Regulvar|http://www.regulvar.com/en/expertises/delta]]/ [[Johnson-Delta|http://www.deltacontrols.com/products/energy-management/energy-management-software]], Lucent.
A group of ~McGill student interns also have access to Pulse Energy Manager, as part of the [[McGill Energy Project|http://mcgillenergyproject.wordpress.com/]]. Some of whom are performing detailed analysis using Energy Manager and its API.
!!!Notes:
''[00:00:03.11] JC'': Department overview. (1) Providing utilties: steam, electicity, chilled water; (2) Managing energy consumption, minimzing consumption, energy consumption projects, collaboration with other units / faculties on campus.
''[00:00:54.08] JC'': 2 major campuses at McGill (downtown main campus, suburban secondary campus); more concerned with day-to-day operations of downtown campus (90%); powerhouse and metering network for main campus
''[00:01:38.05] JC'': Duties as energy manager: energy conservation projects, keep track of savings realized, finances, reporting major component, very important all energy projects involve borrowing money from University which has to be paid back; how much is saved lost determined via Pulse, very much involved with different users on campus, trying to raise awareness re: reducing consumption (no incentive amongst campus users because they don't pay themselves for their energy use;
''[00:02:59.09] JC'': another energy manager (Fred) who is oriented toward relationship w/ building operations (e.g. HVAC, managedby other units); interface between groups
''[00:03:41.03] MB'': how long partnered with Pulse? ''JC'': 3 years; both energy managers use Pulse Energy Manager to the same extent, though JC looks at reports more, while other energy manager looks more at real-time curves [Managenent tab], even though he works with [Reports tab], JC also works with [Management tab] from time to time
''[00:04:22.14] MB'': any other users? ''JC'': Yes, also a project manager on the team who works with it, cooperative work with building operations, a group of students who have applied research projects that use Pulse for a whole bunch of reasons (McGill Energy Project), a team who investigated installing solar panels on McGill residences and making predictions using Pulse; another student (MEB) working on an optimization algorithm to predict steam demand, working with Pulse Management tab; project basis. Access to Pulse and training provided as needed.
''[00:06:42.07] MB'': Share screen, show some of the analysis tasks that are currently done, use as springboard for further discussion.
''[00:07:49.21] MB'': how often are tasks performed in Pulse EM? ''JC'': different tasks, for example, examining steam consumption during summertime (smaller boiler is used during summer), need to check all buildings on campus and their demand so that it doesn't go over a certain level, otherwise a need to start up the bigger boilers and energy savings goals cannot be reached. We check steam consumption often for each building, at least once a day during the summer; different curves and we get into detail as need be; done by JC or Fred (other energy manager); JC opens Pulse every day. "We always have something to look at, always a question that needs to be answered"
''[00:09:26.12] JC'': Example [00:09:32.22]: Red curves = total steam demand on campus, other 4 curves represent campus districts; If more detail needed, would try to zero in on the building that's consuming too much steam at the moment.
''[00:10:04.02] MB'': are districts the same size? ''JC'': in terms of size, they are similar. 2 north districts are medicine and engineering districts, consuming more steam than other districts.
''[00:10:27.10] JC'': we keep track of power demand in the summer, many chillers on campus, trying to manage power demand, to mininmize as much as possible; costs nearly half a million dollars every month in the summer, trying with building ops to keep track of this demand.
''[00:11:12.20] JC'': [switches to chart of electiricity per district]goal is to give feedback to building operations so they can do some peak shavings, to allow us to pay less. [black curve aggregates all districts].
''[00:11:52.04] JC'': dark blue and light blue curves, one is electrical demand on main entrance, other is sum of all buildings; main entrance = one electrical entrance on campus, distributed, compare what was paid to Hydro Qu壠to the sum of the consumption og different buildings, allows us to visualize the losses occurring on the network, or metering errors. But in this case, more losses (2-3%, which makes sense considering size of network).
''[00:12:57.10] JC'': Go into more detail, understand patterns on a daily basis, when it's hot outside (relying on outside temperature).
''[00:13:09.08] JC'': Would love to see in addition to OAT and power demand: relative humidity.
''[00:13:31.21] MB'': One of the Pulse ideas is integrating multiple weather variables.
''[00:14:03.05] JC'': re: Wind speed; we don't currently keep track of it, so we don't know how much it influences weather demand; in the winter, it probably plays a big role; Also, the level of sun, solar radiation / cloud coverage - these could also help us to understand what's happening in the buildings.
''[00:14:31.11] MB'': how are weather variables currently integrated? ''JC'': [switches to a single building]; the way we've worked around [not having weather data] is to train typical curves that factor in these different weather parameters into account appropriately when training the model. When we have meetings with building ops, we show the charts, but we still need to explain why power demand increases as humidity increases, as is currently done with OAT. Possibly also with wind speed. We need to see these different factors plotted against time or plotted against each other to see what's happening. The typical demand is a great tool, but it's still a bit abstract sometimes. A black box, which I understand, but if we can plot at the same time that in this case, temperature is the driving parameter for this building, whereas for tha building it's temperature and relative humidity, it could help as well. If I could plot this, at least I could see where humidity doesn't play a big role.
''[00:16:37.21] JC'': we meet w/ building operations twice a month, Fred [other energy manager] and I look for problems, issues, look at different curvers, try to identify abnormal behaviours, analyze it with building ops: HVAC manager, system programmers, control technicians, etc. [00:17:19.25] The way we do it is we show them this [Pulse energy manager Management tab] on a big screen, we focus, show them trends 2 weeks ago vs what's happening now. Building ops has access to [Pulse Energy Manager] bu t they don't use it from what I know. We've trained them, but they don't have the time; we're the watchdogs, in a way, and every time there's a problem, we use different charts to explain.
''[00:18:13.04] MB'': your daily routine: ''JC'': day-to-day, in the morning, (Fred might do this to, though not positive), so we plot whole campus for chilled water, for power demand, for steam demand, we just check whether there's been something weird over the past 24 hours, or longer if it's after a weekend.
''[00:18:55.18] MB'': and what if there was something weird? ''JC'': Today, there were spikes that didn't look normal; edits [the curve], if there was a spike in steam demand for the southeast district, then I would edit the district, look for which points are in this calculation, and then I will go and check each of the different points to see which one was causing the problem. Another way to do it is to look at all the steam graphs, which is what I was doing at the beginning, 3 years ago, but it took a long time to go through all the buildings and not see anything.
''[00:20:34.01] MB'': because they weren't yet grouped [aggregated] by zone]. ''JC'': exactly, I work backwards, I get the steam demand on campus, then investigate anomolous zones, and then check which points or which buildings are in that zone, and then zero it in this way. If need be, if I see if it is a spike in a particular building and it makes sense, then I will try to investigate with building operations, otherwise a lot of the time, it's because our guys play with the meters, in the sense that they verify the meters; not sure what they do but they send wrong signals in a way and I have to correct it, and other times [anomalies] happen which I don't really understand.
''[00:21:37.08] MB'': do you use the messaging functions to explain anomalies? ''JC'': not so much. I tended to at the beginning, but not anymore, I just correct whatever it is I have to correct or if we have to investigate, we just investigate it ourself, and we'll document the outcome with building operations, so that we have a log of all the anomalies and the solutions. This is not in Pulse but in an Excel spreadsheet and a Word document, shared among energy managers and building ops.
''[00:22:32.03] JC'': Looking back, I should have used messaging system more, the ability to put messages or tags when something happens provides a trace of some sort.
''[00:22:51.20] MB'': How many buildings in each zone? Between 12 and 20 in each zone. ''MB'': In a flat list, to go through each building in turn is inefficient, it could be the last one that's anomolous.
''[00:23:22.24] MB'': are some buildings known to be more erratic than others? ''JC'': definitely, there are the really big buildings that are hard to interpret, medecine buildings, for example, we look at them and we don't always know what's happening.
''[00:23:45.10] JC'': Example: electrical demand of this building (McIntyre medical) - it's hard to analyze, it's easy to tell when we start or stop the chillers, but there's so much research equipment in these buildings, so it's hard to tell what's the HVAC system and what's the rest.
''[00:24:15.12] MB'': Are you doing any long term comparisons? Season-to-season variation? Year-to-year? How often does this occur? ''JC'': yes, but I'd use the reporting module more for this, every day as well. We give this information to consultants (electrical and mechanical engineers). They appreciate it because it allows them to work better. Another way we use it is for billing downtown [campus] clients. I have an little app (developed internally, not really an app but a software solution developed in Access) that calculates the invoice of each of our clients and I doublecheck the results of the app, so I counter-cross what the app tells me with what is in Pulse. I do this once a month only. We have 50 meters to bill on campus (out of 400, or 10%, which make up about 25% of total consumption), we used to do it with Excel, which was hell, now we have a slightly more refined solution. For the other 350 meters, we have other reports.
''[00:27:12.11] JC'': [switches to show McLennan Redpath Library report] we had a ventilation upgrade. This is not the last report I used, we have several people working on it, and we keep on changing what we want to see. The last time I used it I had this compare-by-month visualizaton instead of the past 12 months, and it keeps on changing because everybody uses it.
''[00:27:54.26] MB'': how often do you compare seasonal weather variation in these long-term comparisons? e.g. this summer to last summer, this winter to last winter, etc. ''JC'': Not so much. I tend to do this manually, sort of. With steam, I do this once every quarter for a quarterly report, but it's not fully automated. I don't take everything that's in Pulse. I would take steam [consumption] and McLennan, and I would take this report [month-to-month] and add heating degree days, and then I would compare it using normalization methods or whatnot. We haven't been able to train typical curves for all the buildings, so sometimes we have to do it manually.
''[00:29:16.20] MB'': weather normalization as a need identifed by Pulse clients. ''JC'': yes, this is why I always do it on the side. Also, we have fiound that I don't know how solid the model is. Especially for new conditions: buildings keep on changing, there's tons of projects happening around here. For me it's as valid as jsut normalizing your data to heating degree days in the end, whearas the whole purpose is to have something more rubust.
''[00:30:08.19] MB'': what about home tab visualization? ''JC'': a little bit but not too much. To tell you the truth, I just look at [combined consumption] to see relative consumption of electicity and steam. But the graph below it, energy waste vs. energy savings, I tend to not agree with it, or to me it doesn't really mean anything: energy waste compared to what? Is it compared to typical curves? Because I'm not sure about the [typical] curves, I'm not sure about the wastage and savings. I don't use this one [visualization] very often at all. The home tab I use, but again just to keep track of big consumption but also to keep track of the profile over the past 12 month.
''[00:31:32.01] JC'': Rollovers and roll-ups are killing me. I spend my life changing points and adding roll-ups and rollovers. ''MB'': what are these? ''JC'': when a meter goes from 99999 to 0, it's relatively well accommodated by Pulse, but roll-ups, when something occurs with a meter and the signal is wrong for some reason and you end up having billions of some unit of energy, which doesn't make sense. You have to go to the point and just cancel it, and add roll-up. This takes a lot of time, is there an easier way to do it?
''[00:32:25.16] JC'': [data cleaning] Here for example, steam consumption is not possible. Something I could imagine is that I double-click on it and then go to the points that make up these values, then see which one of them accounts for this huge number, for example, allow me to go what the source of the problem is. [right now it's ambiguous because you're looking at the whole campus]. But that's how I use it.
''[00:33:13.09] JC'': If it [the Home tab] were better set up, not by Pulse but by me, what you have at the top is actually pretty relevant; consumption, consumption per area, savings is what we look at. So I don't use it [the home tab] that much just because I know there are some stuff [the energy savings/waste] that don't make sense.
''[00:33:42.01] MB'': Do you use any management / reporting tab visualizations to visualize your porential savings? ''JC'': I do, a little bit for those buildings [e.g. bookstore], ''MB'': for projecting out? ''JC'': not projecting it too much I think, or could we? but at least is shows the savings [shows bookstore electrical demand with cumulative savings] I use it a little bit, especially in reports, but only for buildings where I trust the baseline or the typical curve.
''[00:34:45.10] MB'': Alerts: use of alerts and visualizations that accompany alerts? Is that functionality used at all? ''JC'': A little bit. Because it takes time to set up, over the past few years we've been working with interns and I've asked them to set up the alerts. I would register for the different alerts but the problem is that it was not well set up and we ended up having a lot of emails, so we just unregistered everything. Recently, we tried again, but used the typical curve or using different parameters. We tried as a pilot on a few buildings only, to see what it looks like.
''[00:35:50.07] MB'': So currently are you receiving alert emails? ''JC'': I am, for one of the buildings. The only I haven't unclicked. Fred, the other energy manager, I think, recieves them for a cluster of buildings but I'm not sure which one. ''MB'': how often? Alerts don't occur very often but when they do, look at visualization generated by alert. We have set up alerts especially for power demand in the summer, so in my case I have one here in my inbox, I thiknk I received one a few days ago. When I view the graph for example, the graph doesn't say too much to me, it doesn't mean much to me. [switches to alert email link and to visualization]. If it were to point to the graph I use, it would mean more, because I could navigate, I could do stuff with it.
''[00:37:41.11] JC'': alerts should point to a chart that I'm familiar with; the alert chart lacks context. It would be more meaningful.
''[00:38:50.10] MB'': threshold messages that show up in these charts [reporting tab]; have you explored that functionality? ''JC'': No. ''MB'': In messages tab, user messages and threshold messages, setup in parallel to alert system you already set up.
''[00:39:45.24] JC'': cumulative data doesn't matter. instantaneous vs. cumulative data. This is the value of reporting tool. demand (kW) vs. consumption (kWh), is there an easy way to go from energy demand to energy consumption? ''MB'': A way to show those separately, linked charts for same time window, linked navigation between consumption and demand charts. Rather than overlay this information on the same chart, but have linked selection and navigation between charts.
''[00:42:16.06] MB'': any other problems with current explortation of data? ''JC'': One thing that would help When you train a typical curve, determine significant parameters. Which one is most significant parameter, so that I wouldn't plot power demand vs. wind speed if wind speed doesn't have an impact. Determining which factors to normalize or factor out.
''[00:44:05.09] JC'': black box and Pulse's proprietary analytics. I realize that the goal is not to understand what's in the black box, as I understand that Pulse wants to protect this knowledge. But it's more to undertand: what are the driving parameters? I want to understand my buildings. ''MB'': Especially in plannign stages, looking at potential savings in the future, to ensure an accurate forecast.
''[00:44:39.11] JC'': Pulse is way more user-friendly. A need for drag-and-drop options, or capacity, for building charts. Something more intuitive. ''MB'': currently, setting up points multiple times. Either something with drag-and-drop or learn from recent selectons. Pulse is improving some of these workflows in the next release.
''[00:46:46.16] MB'': Other tools? ''JC'': We shoped around. Tried [[Rockwell|http://ca.rockwellautomation.com/]], automation whatever-its-called, [[Seimens|http://w3.usa.siemens.com/buildingtechnologies/us/en/building-automation-and-energy-management/apogee/software-and-web-applications/insight-advanced-workstation/Pages/insight-workstation.aspx]], [[Regulvar|http://www.regulvar.com/en/expertises/delta]] / [[Johnson-Delta|http://www.deltacontrols.com/products/energy-management/energy-management-software]] building automation / control systems for HVAC systems; we've seen demos of Lucent technology, which we didn't like. Lucent looked cool but far away from what we do from an energy manager's perspective.
''[00:47:54.04] MB'': Why Pulse? ''JC'': Features, a good tradeoff between intuiitiveness, aesthetics, something we could show to community, something we could work with for a business perspective point of view. For presentation and analysis. Lucent was nice but not appropriate when you want to do energy conservation projects and whatnot.
''[00:48:32.19] MB'': What parts of Pulse are used for presentation? ''JC'': We try to show chart online, zoom in, zoom out, browse from one building to another, compare buildings, compare building to another with different conditions. We do have static versions for 15 minute meetings, just to prove a point but not analyze it.
''[00:49:31.14] MB'': Send out links to Pulse charts before meetings? ''JC'': We've tried. Not successful.
''[00:49:46.15] MB'': Wrap-up.
''[00:50:59.13] JC'': On the overall, a pretty good tool. Contact for McGill Energy Project students.
!!MEB Interview Notes - 13.08.03
MEB is a recent B. electrical engineering graduate and co-founder/project coordinator of the [[McGill Energy Project|http://mcgillenergyproject.wordpress.com/people-2/people/]]. MEB will be on staff for a year, after which time he plans to work i nenergy consumption consulting services.
His recent work involves using Pulse's API to generate accurate predictions of ~McGill's steam consumption on an hour-to-hour basis. ~McGill's steam powerhouse boiler staff are looking to optimize their use of ~McGill's four main boilers, as there is a cost associated with turning on or off the boilers, and the combination of which particular boilers at any given time (they are not of the same output capacity). Insufficient steam output can result in a network outage, which is costly; on the other hand, too much output can result in energy waste.Currently, the powerhouse staff rely on their combined experience and powerhouse sensor outputs to decide whether or not to turn the boilers on/off, but they would benefit from increasingly accurate predictions of steam consumption 2-3 hours in the future.
Originally, it was thought that the information necessary for predicting steam consumption could be acquired from Pulse's typical (benchmarking) curves. However, the typical curve wasn't fine-grained enough to make accurate predictions on an hour-to-hour basis; it was more appropriate for typical curves spanning longer time intervals, weeks or months.
This limitation led MEB to extracting CSV summaries of campus steam consumption for the downtown campus from the Pulse API, where he polls real-time and historic steam consumption data as well as multi-dimensional weather data (recent and forecasted). This data is used to train and update a SVM, which returns predicted steam consumption values for the next 24 hours, generated and updated each hour. MEB is still experimenting with the time window and weather parameters; for instance, humidity is important in the summer, wind speed is important in the winter, and temperature is important year-round; solar radiation is also an avaiable parameter. He is experienced with scripting languages such as R and Matlab, and is using an R wrapper for ~LibSVM, an SVM package written in C. This ~SVM-based predictor has already shown to be up to 40% more accurate than Pulse's typical curve for predicting hour-to-hour steam consumption.
The output of the SVM predictions is then handed over to his colleague, AO (technical specialist), who is currently building an [[web-facing visualization|http://mcgill-steam.herokuapp.com/]] using [[Sinatra|http://www.sinatrarb.com/]], [[Heroku|https://www.heroku.com/]], and Ruby for consumption by power-house staff. They aim to have this simple visualization ready by the end of August, at which point they will demo it to powerhouse staff. I'll check back with them afterward to see how the visualization was received by powerhouse staff and how it affects their decision-making process, and whether any measurable optimization in steam consumption is achieved.
Limitations of this simple visualization (at current) is the absence of confidence intervals. I brought up the question of parameter loadings into the prediction, which is an issue that JC brought up re: understanding Pulse's typical curve. We discussed the notion of sensitivity analysis and how the various consumption and weather parameters might affect the predicted consumption values, and opportunities for integrating parameter information and interactive control of parameters into an interactive visualization. We also touched upon toolkits for interactive visualization on the web, such as D3 and Processing.
MEB is not an active user of the Pulse Energy Manager tool, though he is familiar with it as a result of evaluating the predictive power of Pulse's baseline curves. He remarked that he experienced a significant interaction lag when navigating between points in Energy Manager; this lag should be reduced in Pulse's forthcoming release, which will reduce the need to generate points and navigate between them. Template charts on the new home tab will alleviate the need to generate and toggle between points.
Update 04.03.14: [[McGill Energy Project Map|http://mcgillenergyproject.com/map.php]]
Here are my comments / ideas to complement those of JN and CC, with a focus on the vis aspects. My raw notes are in Dropbox/Pulse/wireframes/Energy Managers for Utilities/Feedback Sessions.
I was intending to add these notes to screenshots of EMU as I had done previously, but for some reason I still can't login to em.ca.pulseenergy.biz (403 error).
@JN, we can discuss these comments in more detail when we next meet.
Cheers,
- MB
NOTES:
Heat map - problems:
- The time scale along the x-axis of the heatmap is very distant from the date picker, and this distance will be greater if/when lazy-loading is used instead of pagination.
- The final date column in the heatmap represents a partial interval, and appears to contain the final day of the previous interval: for example, if the selected range is May 1- Jun 30, column #n-1 is the range from Jun 24-30, and column #n is June 30.
Heat map - possible action items:
- Display the dates along the top of the heatmap in addition to or instead of at the bottom. Alternatively, allow the date axis to remain at the top of the browser window when scrolling down or lazy-loading heatmap rows.
- Ensure that the heatmap columns contain non-overlapping intervals. Partial intervals could be visually distinguished by varying the width of the final column.
Box plots - problems:
- JC admitted that despite learning about how to read the box plots back when I was interning, he still doesn't know what to do with the distribution information.
- The inclusion of the aggregate of all sites was confusing: juxtaposing the box plot for the aggregate next to box plots for the individual sites could lead to invalid questions about why distributions of individual sites differ from the distribution of the aggregate.
Box plots -possible action items:
- Add mouseover events to each box plot such that the values for the 25 percentile, median, and 75th percentile are revealed in a tooltip when hovering over the box, lower whisker and upper whisker values when hovering over the whiskers, and outlier values when hovering over an outlier.
- Add different styling to lower-range outliers and upper-range outliers (such as blue dots and red dots, respectively). JN and I have discussed an idea to also encode these outliers in their corresponding heatmap cells, either as a highlighted border or as an icon within the cell.
- Remove the aggregate row from the heatmap and its corresponding box plot [OR] vertically separate the aggregate row from the main table.
Load profiles - problems:
- JC reported that the focus+context zoom behaviour had been erratic, and not always displaying an appropriate resolution in the main focus window. This behaviour could not be replicated when we spoke to him.
- The y-axis locking was inconsistent when switching between historical comparison and tile view, sometimes the y-axis locking persisted when changing views, other times it did not.
- The site overlay load profile view and site overlay + historical comparison views were unreadable with more than half a dozen sites. We looked at portfolios with 6 sites, 3 sites, 17 sites, and 30 sites. The example CC and JN referred to was part of the 30-site portfolio, though JC had to manually filter out many of these sites before the unexpected behaviour became apparent. JC mentioned that he would prefer to start by selecting individual sites from a list in order to populate a site-overlay view, but this would require starting with a blank canvas, which can be intimidating / confusing.
- Line colours corresponding to sites are not stable between site overlay view, tile view, contribution view. As CC mentioned, JC asked about having a consistent colour scheme for resources (similar to how the current EM portfolio page uses yellow for electricity, purple for natural gas, etc). A potential legacy bias. JC also mentioned that some customers want/expect a consistent colour palette.
Load profiles - possible action items:
- When switching between the aggregate load profile and any site overlay views, it makes sense that the y-axis lock should be not be carried over. The y-axis lock should be retained when using the browser's back function, but not when switching between different load profile views.
- Once implemented, tagging and unit normalization will make comparison of multiple load profiles from different sites more meaningful. However, there still may be cases when more than half a dozen sites are selected. One alternative to the current separation between aggregate view, site-overlay view, and tile view is to display wide-aspect ratio individually-scaled load profiles such as in this design: http://bl.ocks.org/mbostock/1157787. The user could select a subset of these load profiles using check boxes next to site names to view the selection along a common y-scale in a larger plot. This larger plot could display the aggregate load profile by default. The size of this subset could be limited to the number of visibly discernible colours (8-10). @JN: we can discuss these ideas in more detail when we next meet.
Consumption bar charts - problems:
- The y-axis read kW, and not kWh.
- Bar charts grouped by site are difficult to navigate when many sites are visible. The brushing window cannot be adjusted to be any wider.
Consumption bar charts - possible action items:
- Similar to the load profile problem described above, a tile view might be easier to start with than a grouped bar chart, such as in this design: http://bl.ocks.org/officeofjane/7315455. Once again, we could provide an option to directly compare the bar charts of a small subset of sites on a larger chart with a common y-scale.
''Role'': building automation software specialist @ Pulse, has worked with EM users and clients (including UBC, SES, Retirement Concepts), setting up management charts, reports, and baseline curves; instructing users on how to use the new home tab: Plan / Optimize / Verify (POV).
!!Thoughts on EM:
*''EM Users'': Energy Managers often don't have time for analysis, and Building Operations Personnel don't have the skill set necessary for analysis; ideal user isn't a position employed at most organizations, so EM design must address managers and building operations
*''~High-Level'': Energy Manager (EM) should offer interpretation and suggestions, not conclusions, identify ways to integrate these suggestions into EM / chart interfaces: example, SP of demand vs. OAT, with inflection points between 13d 18emperature range where chillers and coolers may overlap)
*''Tags'': any database field describing a building should be a tag on which to facet the data and display for side-by-side comparison; charts (primary use, geographical location, occupancy, year constructed, area); further goal to allocate parts of buildings to tags (such as UCB use case of allocating parts of buildings to departments). A need to compare within and across these cross-cutting descriptions of buildings.
**''Example use case'': community pools (sometimes co-located with rinks, gyms, meeting rooms, 련ow to separate these out to make legitimate comparisons?
**''Example use case'': Retirement Concepts client has facilities spread over BC interior and Southwestern BC; faceted comparison by geographical area can reveal outlier (building in Williams Lake)
*''Energy allocation'': Pulse Check recommended energy allocation (outdoor lighting, water heating, യ become metadata fields in EM; assists in attributing power consumption to particular sources (when setting up a buildign in EM, these are entered as database fields with autodetected / suggested responses
*''~CUSUMs'': single building or multiple buildings, no current way to select subset of buildings
*''Validated Savings'': was savings due to Energy Conservation Measure (ECM) or to changes in occupancy, other reasons? What is cause of inflection point?
**Attributing savings to particular events / ~ECMs, facet by validated and invalidated savings, comparing before / after ECMs
**Baselines based on multiple regression, considering validated ECM measures only; need to subtract effects of other events making baseline invalid
**ECM messages in EM currently a visual indicator only, doesn't affect baseline calculations
*''Multiple baselines'': tagging baselines (valid / invalid), recommending alternative baselines with associated noise / uncertainty
*''Noise'': quantify noise from from baseline calculation, visualize that uncertainty (see band visualization from mockups sent by Lillian. Z), standard visualization, regardless of type of baseline
*''Multivariate rankings'': relative and absolute comparisons of buildings in a portfolio, ranking by energy intensity alone (as in Marc T.࣡se, may not be best ranking); ability to rank by multiple variables, include uncertainty in ranking
**SP of intensity vs. absolute savings for rankings?
meeting w/ KN, energy analyst @ UCB via Skype + screencast recording:
''[00:00:03.04] KN'': Role: energy analyst, sub-metering at the building level.
''[00:01:17.29] KN'': Dashboards for outreach, marketing.
''[00:01:38.08] KN'': role on back-end. Responding to questions re: dashboard anomalies. Initiate investigations.
''[00:02:40.06] KN'': to do: benchmark the buildings. Normalized comparisons with other universities. [[Labs21 benchmarks from Livermore national labs|http://www.labs21century.gov/]]. Most available data is commercial data, not for campus buildings. e.g. portfolio manager.
''[00:04:10.27] MB'': comparing regular buildings and buildings with special equipment
''[00:04:20.28] KN'': very granular, at the university level - target for universities is are we doing OK relative to other universities. how do we want to set 10 year goals for energy planning, greenhouse gas
''[00:04:50.14] KN'': gap between energy information systems and building automation systems / EMS. Figuring out what the spike was, a way to bridge that. You can actually do that with Pulse. Detecting energy leaks, suggesting what might be the cause.
''[00:06:12.01] MB'': How long have you been partnered with Pulse? ''KN'': a year and a half.
''[00:06:23.11] MB'': how much of day-to-day-work done in Pulse? (KN shares screen).
''[00:07:07.01] KN'': Uses Pulse on a weekly basis, Monday or Tuesday. First thing noticed is that not enough of Pulse's buildings are baselined (top left), this is ignored. "Savings" at top of "Home" tab also ignored, because not all buildings are in Pulse, savings relative to data housed in Pulse, and may not be representative of full portfolio.
''[00:07:56.11] MB'': energy savings goals are larger than just the buildings reported on in Pulse
''[00:08:06.24] KN'': only 3 buildings making up the 3.9% savings, so global metrics are ignored.
''[00:08:19.10] KN'': uses Management. starts by checking a few of the big buildings. Begins with total electrical demand overall for campus, for Electrical demand and steam usage.
''[00:08:52.11] MB'': is UCB all one campus? ''KN'': yes. 90% of buildings concentrated on core campus
''[00:09:10.27] MB'': 2 curves shown here. Not baselining yet. ''KN'': Right. Checks religiously, checks campus first, then the big users on campus, a group of buildings that are most energy intensive. (6 curves, 6 different buildings many threshold events on top curve, zooms in on Aug 12). "I look for trends, or rather not trends but anomalies." Determining what is worth looking into. (Hovers over threshold event on top curve). Threshold alarms set up for Stanley hall (top curve), set up with Brent and Harish, anytime it goes above 2 megawatts, we get an alarm, on the software, not via email.
''[00:11:18.07] MB'': how often is threshold triggered? ''KN'': For Stanley hall, quite often (would be a lot of emails, wouldn't want to deal with it).
''[00:11:32.08] KN'': Also compares libraries, mid-buildings, large buildings.
''[00:11:42.10] MB'': how did you decide on this threshold? ''KN'': At the time, we threw this number on the table, 2 megawatts, we want to be involved. "Something pulled out of thin air."
''[00:12:00.29] MB'': and you would then send building operations to investigate the building? ''KN'': not quite, this happens quite a bit. (brings up Stanley hall electrical demand curve, 6 month view). We initially started dispatching people to go look into this stuff, now having a better feel for these building and its sensitivity to OAT, this one day had 4 alarms, this week had 10 alarms; I've grown a little insensitive to the alarms now, because I know when it surges over 2 megawatts it's because it's a hot week in Berkeley.
''[00:13:06.15] KN'': for me, it's an added level of work because I have to do this (compare threshold cluster to OAT) just to understand that the building is using more energy because it's a hot week.
''[00:13:39.18] MB'': what type of building is Stanley hall? ''KN'': Stanley biosicences facility, the largest energy user on campus. about 14 million kilowatt hours annually, so that's why it's on Pulse and I pay attention to it. It uses a lot of outside air, so it's more sensitive to temperature variation, as opposed to some of the other buildings, so this is one analysis I do.
''[00:14:14.01] KN'': If I get an inquiry / complaint from the outreach time (example; Haviland building), someone might email him after noticing an anomaly on the dashboard. ''MB'': a user of Haviland hall? ''KN'': could be anyone. Usually from a building manager, or staff inside the building, researcher, technical facilities personnel.
''[00:15:06.28] MB'': How much of analysis is inquiry-driven? How much is top-driven? ''KN'': good question, limited resources, diminishing returns, I look at high energy users, and when you go to a small building (example: Anthony hall, 0.5 to 3 kilowatt building), (admits that this is the first time looking at this building). I tend to look at big buildings, anything that doesn't make it on his kW charts, the big buildings, I won't go out of my way to look at these small buildings (e.g. Anthony hall) every week. I might look at it once a year. Those investigations are mostly driven by complaints, feedback by customers.
''[00:16:25.20] MB'': how many complaints / feedback requests in a month? ''KN'': handful, no more than 5. the point of outreach campaign, at some point, people will look at Pulse dashboard when it's new and the campaign has just begun, more of a marketing issue. Example: when Barrows hall's [dashboard] first went up, we got about 3-4 emails in first month alone. Nowadays, no one really cares, partially because users are not as educated as we are to what all this squiggly means.
''[00:17:25.00] MB'': How far back does data go? ''KN'': usually for a year, max, that was the time. starts ticking as soon as meter is installed. We've talked with Brent, there will be a concerted effort sometime in the future to actually upload data into Pulse for historical usage. End goal is to use Pulse as invoicing software. The campus has about 100 buildings, and we've divided that into divisions, or by departments, each departments has its own portion of a building, with the new Berkeley incentive program, we are dividing up buildings by square footage and apportioning an equal amount of electricity use to that square footage amount and sending an invoice or statement to a client or statement every month. If they go above a baseline, they pay us, if they go below, we pay them, that's the incentive program.
''[00:18:59.12] KN'': This is currently implemented, but not in Pulse, because not all buildings are in Pulse.
''[00:19:07.29] KN'': Goal would be: we might have data for a year, and a baseline year, we'll have to work with Brent to define a baseline period, in this case campus has chosen to use fiscal year 10-11 as a baseline. We're locked into those numbers.
''[00:19:33.24] MB'': this [baseline] data currently isn't in Pulse? ''KN'': No. We don't enough data for those buildings, back then without the smart meters, all we had was monthly meter readings. At the least we have data at the monthly level. Not the same granularity. All we can compare is month-to-month.
''[00:20:04.00] KN'': To some degree, clients are not involved, all they care to see is month-to-month, the clients are not building engineers, they're deans, CFO, or CEO of a college or department, they tend to care more about dollar amounts and trends and performances, savings and overage. We're okay with that, having that level of granularity, but when it comes to technical analysis and energy work, at the minimum this [indicates day-to-day time-series chart of Barrows hall's electrical demand], this is what we want. The best would be to regress this with temperature.
''[00:20:57.20] MB'': are there plans to put remaining buildings into Pulse? ''KN'': yes. we're putting up a proposal. Have as many buildings in Pulse as possible, along with as many commodities as possible: water, steam, natural gas maybe.
''[00:21:26.08] MB'': currently only electrical demand? ''KN'': yes, electrical only. [shifts to home tab] Actually, there are 13 meters where water is tracked.
''[00:21:56.25] KN'': disconnect b/w EIS and EMS systems (Siemens, Honeywell). Stanley hall a good example again, We did a training, Pulse adaptive model established (orange lines). What would be ideal to use the Pulse model as an alarm feature, currently threshold alarms are hard-coded to 2 megawatts, but it would be better to have an alarm based on Pulse model.
''[00:23:15.02] MB'': how many months / what time window was used to create benchmark? ''KN'': I don't remember, I don't remember how long of a training period has, at least 3-6 months of data. Able to see some seasonal variation.
''[00:24:07.14] KN'': It would be good to get alarms based on Pulse model instead of these hard-coded threshold numbers. Both are helpful, so that's a challenge going forward.
''[00:24:31.04] MB'': are you visualizing differential between actual and benchmark? ''KN'': No, ideally that would be in our reports to our departments going forward. Example: if you're college of chemistry, you have 10 buildings in your portfolio, this is how they're doing, this is their variances relative to baseline that we defined.
''[00:25:07.17] KN'': Something like this [RSSP monthly report], except instead of energy (kWh on y axis), it might say delta, and colored bars would be different buildings.
''[00:25:27.24] MB'': these reports are exploratory? ''KN'': yes. Not sent out to anyone, test / technical reports. Like this one [SRB1 building], I attempted to do a regression, and realized that it varies quite a bit depending on how your define your data points, whether it's 15 minutes, 30 mins, an hour, 2 hours, you can get different lines.
''[00:26:06.06] MB'': what are we looking at now [scatterplot of energy (kWh) x OAT], days? weeks? ''KN'': let's see [hovers over points] 4-hour interval temperature-energy (kWh), there's a limit to this: Pulse can only crunch up to 256 data points, a limit to data points. You lose clarity when you bundle up data into four hours, because building varies by season. If I picked this [date range point option of past 127 four-hour intervals], if it was a winter vs. summer month, it would make a big difference.
''[00:27:00.13] MB'': this is 7 months worth of points (as indicated in the editing tab), difficult to spot seasons? Currently? ''KN'': yes. range is January - July 2012. I defined this because this building had a major HVAC upgrade, and I was looking for baseline number, because I only did it for 7 months. When I did this I had to define a report, add a report, add the points. It would be grat to have that simplified on to [the management] tab.
''[00:28:11.10] MB'': from what I understand, workflow issues such as this are being improved. Version currently being rolled out at Pulse will have default charts, like the energy vs. OAT scatterplot. Default reports generated for each building.
''[00:28:51.12] KN'': That's fantastic, I give them [Pulse] kudos for working on that. It takes a lot of time.
''[00:28:57.07] KN'': Not as fluid in IE rather than in Chrome, zooming in and zooming out. It would be great on the regression too, on the temperatures, if they had the same function, If I zoomed in on a summer month, it would generate a new regression for summer only, that would be really powerful. So I can look at the two months before a major HVAC overhaul was done vs. the 2 months after. At least I could look at the equations and see that the baseload dropped by 10%, that would be helpful.
''[00:29:43.02] KN'': would be really useful. Example: [Google weather], I use this every day to determine how I should dress for work. It would be helpful if Pulse could incorporate something like this. To get a daily and monthly forecast for OAT, and have that be on the front page somewhere. Kind of like a to-do-list or personal checklist. You could have something here that says "coming up this week, we anticipate temperatures in the 90s for the whole week, and given the analytics / regression and everything else, we anticipate that Stanley hall will be up by 10%".
''[00:31:10.00] KN'': Another one would be a another text box that talks about fault diagnostics, to talk about changes in a building that Pulse notices (''MB'': as opposed to changes that you are aware of, like overhaul of HVAC systems).
''[00:31:41.05] KN'': My ideal would be a section on [pulls up Bing search on Mint performance-to-date charts]. In Mint, it has a pretty cute function that tells you, "hey, watch your spending, your gas budget is almost at 100%". In a token, to have something similar here [home tab margin], independent from a forecast, that gives a summary of what Pulse learned about your buildings. Like if it's running in the background crunching numbers "hey we noticed Stanley hall's baseload increased by 2%, or Anthony hall, it used to be 3kW, we've noticed that in the past 2 weeks or 2 months that it's steadily gone up to about 10kW", like a threshold alarm for more than 100% increase.
''[00:32:55.11] KN'': Another idea which is rally interesting, hasn't seen done anywhere else: a weather sensitivity analysis. So you would run your correlation with OAT to kW and maybe steam BTUs for all the buildings. You see if it has a higher R^2 value, you notify the user, indicate that the building is highly correlated with OAT. ''MB'': and "if OAT were to change by 5젴his is what would happen". ''KN'': yeah, that's the 2nd layer of analysis, kind of like the [Google weather forecast], based on forecast of +70ather, we anticipate building being up by 3%.
''[00:33:55.03] KN'': What I envision this [home tab] to be from an energy manger perspective, from the way I use it, I get the data analytics in the background, but what I'd like to see an even higher-level reporting than what we have here. Kind of like your stock portfolio. You sorta kinda have it here [home tab rankings] in this part.
''[00:35:02.23] KN'': [Google Finance example], my inspiration comes from this, to be honest, it's exactly what you have here [home tab performance ranking]. I do care about consumption and consumption per area, these are good metrics, but perhaps a higher level aggregate like this [performance breakdown], where it shows how many of your buildings are doing great/poor. I imagine you clicking it and revealing a drop-down menu that shows performance of buildings part of that aggregate. I anticipate something like that would be nice. A summary report on this level would be good.
''[00:35:55.07] MB'': yeah, a way to drill-down. Right now it's done by resource, and then by groups of who's savings and who's not. Imagine a way that would lead you immediately to those results.
''[00:36:09.03] KN'': Yeah, I would come in and say, who's my worst performing. For example, if Stanley hall was at -10%, I would check consumption per area, that's something I'd pay attention to, while others at low consumption-per-area would be of little interest.
''[00:36:31.14] KN'': At a higher-level summary table, what I had in was [Bing image search for 'carbon dashboard'], Me : Similar to what you have, as a dashboard experience, a custom view for yourself. Depends on who's using it and what their workflow is, perhaps just a specific set of buildings, might mean a different dashboard.
''[00:37:11.15] KN'': Something like this [Bing image search dashboard], Pie chart would be nice.
''[00:37:22.14] MB'': A few minutes ago, you mentioned weather and sensitivity analysis, to what extent are you referring to a need for interactive simulation of weather variables? Example: adjusting the OAT by a small amount and seeing predicted effect on energy consumption. ''KN'': not quite, you talked about 2 separate things there, first one is what happens to energy use when you change OAT. That ties into this [Google weather] If I have historical data and a Pulse model and it can tell me what my building was doing at all those temperature points, if I were to give [Pulse] a command to say the next five days will have this OAT, given all these numbers, this forecast, what do you think my building will be doing? The answer is different from Pulse adaptive model.
''[00:38:48.16] MB'': I've seen after speaking to another energy manager who hard-coded in these points, create a threshold alert whenever the demand exceeded that by a set amount, wouldn't be using Pulse adaptive model.
''[00:39:09.27] KN'': exactly, that would be perfect, because when you look at today, I can't predict what the next point will be, although it would be nice to predict it. Where I draw my inspiration from is from [California ISO, today's outlook (http://www.caiso.com/Pages/TodaysOutlook.aspx)], doesn't have OAT, it does the modeling in the background. It has hourly time scale (x axis) and power (y axis), same as in Pulse. the thing with the real-time feed and with the Pulse adaptive model is that it seems to be doing it regressively, I can see backwards and see where it should have been vs. where it is, but I like to see what I think the building should be doing given this forecast [back to Google weather], which is exactly what [casio.com] has done here.
''[00:40:29.00] KN'': This is [www.casio.com], California's electricity consumption. You have actual demand, which you have in Pulse, a live feed of data, a day-ahead forecast and an hour-ahead forecast. You can see the next 23 hours at midnight. You can do that once you have a Pulse model and a weather forecast.
''[00:41:38.24] KN'': Sensitivity analysis: [switches to electrical demand of Stanley hall], If all my buildings have a blue line and orange line, (actual and typical), over some defined period, let's say a 3 week rolling window, if it notices that your energy use correlates better with OAT, I'd like to get a warning about that, or if it's going the other way, if it no longer correlates, I'd like to know that too. Reason for this is on an application level, what that tells me as an energy manager is that there's been a change in the building, there was some significant event in the building that changes performance, that event could have been a new chiller install, or a chiller malfunction, or a bunch of people decide to bring in their own space heaters, that's what it tells me. Very important for us to put our arms around campus energy use. Because on a practical level, we've got 10 million square feet and 50,000 people out there, and all we have to relate to them and their facilities is through a bunch of graphs. That's my challenge. I love to learn about it, preferably through graphs. If they can tell me that something's happened in Stanley hall, I can physically go out there and looks at it.
''[00:43:52.01] MB'': question of understanding differences in weather and how that factors into these baseline conversations, and greater clarity as to what's being factored into Pulse's adaptive model. Other energy managers have remarked that it's like a black box, it's hard to know what's going into it.
''[00:44:19.20] KN'': [performance ranking table on home tab] is useful, but maybe it deserves its' own tab, perhaps 'details' or something. On the front page I'd like to see the top 5 performing in terms of performance and the top 5 worst performing. Just a quick summary, flip through this real quick like you would the classifieds, a weather prediction / outlook, a sensitivity analysis, maybe a pie chart.
''[00:45:03.07] MB'': How many buildings will you eventually have in the system? ''KN'': between 70-100. ''MB'': yeah, so going through the [performance rankings] table would take some time, but a more detailed ranking of what's going on here currently in this table format.
''[00:45:30.09] KN'': Right, that might tie back to what I mentioned earlier about [the performance breakdown], electricity as a resource, and I click on it and it drops down to show me the top 5 performing and worst-five performing. That's very practical.
''[00:45:47.24] KN'': [home tab, Overview] Right now we converse in energy units (kWh), maybe this could indicate a dollar amount below it, a minor thing. It would be easy to then share this with senior management, especially if the report compares baseline to actuals, savings numbers, and have a dollar value in real time. It would show what is saved this month.
''[00:46:32.18] MB'': are you creating reports to that effect currently, visualizing prospective savings? ''KN'': yes. I do a lot of this in Excel. Exporting from Pulse into Excel. Every month, I do this, I want to get this months' consumption data, exporting a full month at 15 minute intervals from 12 to 12. This is literally what I do. Import into Excel, take sum and paste into different spreadsheet (loads spreadsheet), what I export from this table for management is here (shows several summary charts), a high-level summary view of CUSUMs.
''[00:48:55.19] MB'': You wouldn't create any CUSUMs at all in Pulse? ''KN'': Right now I don't do it, because I haven't explored it and because we don't have all the buildings in there. There's also another layer of complexity given how Berkeley has chosen to run its program. ''MB'': some of this data from Pulse, some from [monthly] meter reading. ''KN'': yes, and another layer of complexity because some readings are corrected following complaints, so that buildings are credited by some amount of kWhs because some guy from my department accidentally turned on a VFD or a fan for a whole week, so the client shouldn't be billed for that, an energy office error. We credit them with some number. That's what this [highlighted cell and formula in Excel] shows. So there is some data editing, and there's a bunch, with notes, and colour-coding. This is how I keep track internally.
''[00:50:24.06] KN'': All this data with the edits, and there's also SEP adjustments. There's a group on campus that's doing whole building retrofits, and we don't like to credit a client for those savings. So I manually adjust those things out. I go to each department, work with the graphs, I take out a certain percentage. When the client gets this report and it says OU year-to-date estimated savings, distinguishing their savings without savings caused by retrofits, it's netted out to reflect what we think the department did, and that's where this chart comes in (in stacked bar chart: separating OU savings to the department (yellow), SEP savings to the capital program (light blue), and actual (blue), next to baseline bar (green).
''[00:51:13.05] MB'': ideally, you'd have this chart in Pulse or is there enough reason to do this in Pulse? ''KN'': ideally in Pulse, because it's centralized. Right now I do this manually, take manual readings, calculations manually. If it's all in Pulse it should speed up my effort, instead of waiting a full month before getting last month's data. We could do this within the first 5 days of a month.
''[00:51:47.14] MB'': and with charts in Pulse you can add in user messages, just like the message you just showed me about the building being credited, to be able to add those manually to Pulse.
''[00:52:00.15] KN'': Right. That's what this [Excel chart] is showing. Though I haven't seen Pulse pull up anything like this before from a data visualization standpoint. ''MB'': to be able to break things down into categories, attributing various savings. ''KN'': right. It's a complex thing to do in Excel actually. You can't do a stacked chart in Excel like this that has two bars, it's a bit of a hack.
''[00:52:57.03] KN'': These [3 Excel charts] are the the main charts I share with senior management, sent out on a monthly basis.[1. actual vs. baseline electricity, SEP included, meter-to-meter readings)]. ''MB'': the baseline is not from Pulse data but from 2010-11 fiscal year? ''KN'': yeah. It would look like this as an ongoing [chart, deletes data from 3 cells corresponding to 3 months], would look just like this every month. As I go I start populating [the cells] and you get the full 12 months. [ 2. percentage of electricity reductions savings (SEP included)] and this one [3. cumulative energy savings], the best feedback I've gotten is on this CUSUM chart, because a the management level you're given a target to hit at the end of the year, and you're slowly building up to it.
''[00:54:18.11] MB'': but these [Excel charts] aren't going out in to future months, are they? These are just comparisons with to-date data? ''KN'': yes, we run a fiscal year, which is which it's only 12 months. Has a trial period in there [Apr - Jun '12]. For next year, there won't be a trial period, or I might show the next 12 months out here [continuing toward right of CUSUM chart, 27 months total]. ''MB'': Is this 2 year historical or 1 year historical, 1 year predicted? clearly indicating actual from predicted? ''KN'': Yeah. So this is more a high-level summary.
''[00:55:18.12] KN'': One thing I'm not sure if Pulse does this in their monthly reports to clients. Let's look at libraries. [opens Library(2) tab in Excel document]. This is what goes out to a client, because this is higher level, it's all letterhead, a formal letter. Actually, [pulls up PDF version] this is what it looks like. Blah blah blah, yak yak yak, this number [highlights number in text], changes with the data, and some canned text depending on their performance. Ideally Pulse could auto-generating something like this. Right now, I can only afford to say, "you're doing great or you're doing bad", at the aggregate level, of all your buildings, when you scroll down, this is where you get the building level data [3 columns, baseline, actual, percent savings, cells in percent savings column colour coded scaled to amount, bivariate diverging colour scale]. Ideally, high-level message will also flag anomalies at building level.
''[00:57:22.27] MB'': to some extent, this [table] mirrors what you see on the home tab of Pulse. ''KN'': yeah it is. This is a summary high-level, it's colour-coded and this [214% increase, coloured in red] is screaming at me. Then you have a standard cover sheet, information fact sheet, a who's who, a what-do-we-do, what are our programs.
''[00:58:20.29] MB'': so a considerable amount of your time each month is spent generating these reports? ''KN'': Oh yeah, it eats up about 2 weeks of my time to do this. ''MB'': part of each week is about regular analysis, the drill-down sessions, responding to occasional requests, and then a big chunk of your month is spent generating these reports. ''KN'': it does, and the other report which I'd like to see and Pulse once we've got all the buildings [in Pulse] and broken them out by department, is that I'd have some review of the departments, so here you have [in Excel, OU summary (adjusted for SEP), bar chart of kWh savings by department]. Basically you take the same dataset, you slice it first by building, and then by department.
''[00:59:34.03] KN'': this [in Excel, OU monthly savings, grouped bar chart, 1 bar for each month for each department] is monthly, and then I do this by building too [shows Excel bar chart of energy savings by building. Same dataset, only this is by building. And here is the same chart, only blown up by month [grouped bar chart for each building]. ''MB'': you're not sending this one out are you? ''KN'': Well it actually goes to management because they want to see it. I question if anyone would actually pay attention to this, it's almost impossible to read. ''MB'': from what I've seen you can build charts like this in Pulse, though it would probably take more time than someone who is experienced using Excel, could probably generate a lot faster. ''KN'': yeah, and once you have it as a template you can add every month.
''[01:00:45.00] MB'': from what I gather, building templates will be faster, workflows will be faster. ''KN'': right.
''[01:01:24.07] KN'': I'm not sure this makes sense to interpret the data that we have, but this was useful for identifying spatial information relative to light output in a place. ''MB'': what do you mean by spatial, like a map of campus? ''KN'': yeah but this is a map of a table, maybe 50 feet by 100 feet, and we were measuring light output, and these are light intensity maps for different types of lamps. ''MB'': Large portfolios, geographically spread portfolios might be well-served by map-based overview. You might be interested in seeing buildings grouped by location. You may want to see your buildings represented that way rather than as a table, a heatmap may be appropriate.
''[01:03:31.23] KN'': [browses to Berkeley sustainability site], we have the same idea, we have a GIS layer. Some guys attempted this, they were students and didn't get to finish this before graduating. We have a map and you can active a GIS layer over this. And if you click it [McCone hall, a building], it might give you a little bubble that says "McCone hall is 2% up over baseline". ''MB'': Yeah, that's one of the ideas that's floated around after speaking with a few other people. It might be nice to navigate an entire campus this way and see who's performing well and who's not by these means.
''[01:04:36.08] KN'': [browses to email, http://mypower.berkeley.edu], not so much for energy, but rather for sustainability.
''[01:05:31.04] KN'': that's all the thoughts that I have.
''[01:05:35.04] MB'': Another question about weather. You are just looking at OAT, any other weather variables? ''KN'': so far only OAT. ''MB'': I noticed at upper right [of Pulse home tab] that one building is missing a weather station. Is the weather station on campus and does it only report OAT? ''KN'': we do, I'm not sure if it's proper weather station, we might just be pulling this off on HVAC equipment that has OAT. I think we have two, let me show you that real quick [browsing Energy manager].
''[01:06:18.11] MB'': humidity, wind, precipitation, those are not considered currently? ''KN'': No. that's what I meant, I don't think we have a real weather station, I think we have a OAT temperature sensor somewhere, and that's how we have two temperature readings here [Stanley hall], one is at Tolman hall, on NW corner of campus, and this is probably a sensor off of the HVAC. The other one, built in here by Brent or by the group because they found a weather station, maybe at Lawrence Berkeley labs, which might be a real weather station. ''MB'': And that weather station might be able to provide other weather variables. Are you interested in the effects of these other variables? ''KN'': It is. Wind speed maybe, I've never studied wind speed and their effect on energy use that much, but I think it will have some correlation, but precipitation and humidity yes, that will be one. If this is indeed a real weather station it might be just a matter of pulling the rest of the dat points to it. But as far as that weather outlook [switches to Google weather], I think OAT is good enough. I don't think that's really helpful in terms of that analysis we talked about, because it's a projection anyway for the analysis. The outlook's temperatures might be good enough, but for internally, the others may be useful. Precipitation might be useful.
''[01:08:39.06] MB'': what about factoring out weather and making seasonal comparisons or year-to-year comparisons by normalizing for weather? Is that something you do in Excel or is that not currently done? ''KN'': That is done in Excel for select buildings only. Under the [Berkeley] Incentive program we don't account for weather variations. The argument is is that Berkeley's weather is pretty temperate, 50s, 60s, all year round, sometimes in the 70s but that's really it. We don't have a lot of swings. and the fact that we're so close to a large body of water, the bay, that keeps it pretty mild all year round.
''[01:09:27.12] MB'': Yeah, example at McGill between 80 degree weather in the summer and -40 in the winter. Huge swings. ''KN'': we're lucky not to have that.That's where the other tab you were talking about with the weather regression which would be useful just to do that.
''[01:09:55.07] KN'': What I've done with Stanley hall in my own analysis, to figure out what building is doing, I have a whole folder for it, and this is how much work I do for one building, to get this.
''[01:10:15.26] MB'': and this is purely for analysis or do you share/present any of this? ''KN'': This is work and presentation because we're working with a client and they use different chillers in the building, one uses steam and the other electricity, and we've got to match the baselines to the current year. [pulls up spreadsheet]. ''MB'': And these are normalized, removing temperature? ''KN'': Yes. You can do pretty detailed stuff in spreadhseets [browses Excel document, finds correct Excel doc].
''[01:12:33.18] KN'': You do normal plots and residual plots, to which I don't have here any more. Excel didn't save that version. ''MB'': in effect you'd be looking at the residual factoring out weather for this building. ''KN'': yes, so that's what I do manually. This whole year I've only had one request to do it. ''MB'': not something done frequently. ''KN'': Although at some point it would be nice to have that even within Pulse, so I could use it for reporting to senior management. ''MB'': on the fly, rather than custom reports in Excel. ''KN'': Yeah [located residual charts], here's some with charts. I don't even bother cleaning this up for presentation, I just send them this sheet and say here's stuff, go look at it. And then histograms to check for tendencies, pretty random, that's kinda what I do.
''[01:13:43.00] KN'': While we're on the topic of data visualization I had a question for you. [opens Excel with cells coloured using bivariate diverging colour scale], each column is a heating valve ID, each cell has a position of a valve [values 0 - 100], so 100 being fully open, rows are valve positions on every day, 100 = red, 50 = white, 0 = green. We've trended the whole year for about 400 valves. ''MB'': are these mapped to particular buildings? ''KN'': Yes. this is Stanley hall again. The only best way I could think of visualizing this was to sort it and colour code them in Excel [zooms out]. You have a bunch of valves here in the green, never open. And a bunch of valves in the red, fully open. I'm not sure if there is a better way to visualize this, if I were to boil this down into a simple pie chart, or bunch of charts in one page 8.5x11, what would be the best way to summarize all these data points? ''MB'': Spatial position important? ''KN'': no. each row is timestamp, one day. ''MB'': Already by zooming out, you have your own summary already, looking at this heat map. ''KN'': What if I wanted to know which of these are mostly heating? ''MB'': Are some of these valves qualitatively different from others? ''KN'': Only because they serve different areas of a building. The valves are the same. What if I wanted to focus on only the summer months? ''MB'': right, because some times you'd want to filter rows, other times you'd want to filter columns. Small multiple charts / faceting of monthly data? facet by date range or by type of valve or use of valve, or by part of building. Do you use R at all? ''KN'': No, I tried, I just couldn't program. ''MB'': you could probably do something like this in Excel, but it would likely take a long time, and wouldn't be reusable. Learning curve with R. Do you have any programming background? ''KN'': No. Most I've done was in Excel. ''MB'': Also consider Tableau, custom reusable visual analytics templates, generating small multiple, faceted charts.
''[01:20:33.02] MB'': [wrap-up], going forward. consolidating and prioritizing requirements. getting further feedback. Have an input as a user.
''[01:24:01.22]'' call end.
Meeting w/ [[LZ, UBC Climate and Energy Engineer|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]], Sustainable Building Science Program, CIRS 3rd floor, July 9, 2013.
>//oversees the implementation of the Climate Action Plan and is responsible for campus greenhouse gas reporting. She also manages projects that conserve energy in buildings//
>
>- bio from [[Sustainable Building Science Program|http://sbsp.ubc.ca/]] website.
!!Interview Notes
''Context'': LZ is involved with UBC's [[continuous optimization|http://www.projectservices.ubc.ca/portfolio/alternative-energy/continuous-optimization.htm]] program, overseeing [[building tune-ups|hhttp://sustain.ubc.ca/campus-initiatives/climate-energy/building-tuneup]], which are in themselves 3-4 year phased projects.
For each building under consideration, an energy efficiency consulting team is brought in, such as [[SES Consulting|http://sesconsulting.com/]]. First, they make several recommendations for changes to a building's infrastructure and programming. Next, baseline data is collected over the period of a year for the building. Following this, some/all of the recommended changes are implemented. Finally, the tuned-up building is monitored using an EMIS (Energy Manager Information Service, such as Pulse's //Energy Manager//) for the year following the tune-up, in which the improvements are evaluated based on cost and energy savings to see if goals were reached. Anomalies and possible improvements to the tune-up are also identified during this period.
''Usage of Pulse's //Energy Manager// at UBC'': UBC is partnered with Pulse as part of the federal Science and Technology Development Canada (STDC) program, and its use is mandated by BC Hydro. According to LZ, many of UBC's Pulse users have abandoned use of the system, though it is hard to tell who is logging in and accessing the system since there is no accounting or reporting of user activity (who's looking at charts, leaving user messages, creating, editing, or deleting charts or messages).
Currently, there may be only a small handful of active UBC users, and their roles and extent of usage may vary. These include SES consultants, building operations personnel, and continuous optimization planning personnel such as LZ. Usage may change after Pulse's upcoming summer-fall release, which will apparently improve workflow issues (discussed below).
Over the past several years, these projects have entailed a fair bit of organizational shuffling between the sustainability group and building operations. UBC Building Operations did recently hire a full-time employee in an analyst role (BE), one who would be immersed in Pulse's //Energy Manager//, but this individual's job description changed after hiring and there is still no full-time analyst. Currently, Building operations personnel will only look at Pulse when they respond to a power or Heating, Ventilation, Air Conditioning (HVAC) maintenance request; they will not set up charts or reports. The skill level of //Energy Manager// users having access to the UBC portfolio varies tremendously. Setting up charts and reports is too difficult for some. SES consultants may set up charts/reports. LZ thinks that these differing roles might benefit from role-based personalized dashboards, appropriate for an individual's skill level and job duties: energy manager, consultant, building operations personnel, analyst, continuous optimization planning committee.
''LZ's //Energy Manager// Usage'': LZ sporadically uses the //Management// features of the //Energy Manager//, seldomly using the //Home// or //Dashboard// features. Her use depends on the phase of a building's tune-up. She wonders why the //Reporting// tab is separate from the //Management// tab; from the //Energy Manager// help documentation:
>//The Management tab displays live, real-time data on the selected chart. You can also view historical information.//
In contrast, the //Reporting// tab is described as follows:
>//Pulse provides reporting on financial, energy, and environmental performance. You can compare buildings against their typical performance, other buildings, or historical performance on costs, energy, greenhouse gases, and more. Each type of report gives you insight into different aspects of your energy performance.//
''Management Charts'': two types of charts frequently accessed in the //Management// tab are time-series charts for buildings undergoing / having recently undergone a series of tune-ups. Examples include the CEME ([phase] 1), or Buchanan Twr (P). These charts have a consistent naming convention (building name, phase, resource type, chart type), and thus are easily found using the Management tab's search function. Charts are listed as a flat list in alphabetical order, as opposed to a hierarchy of according to chart author, resource type, or building.
These time-series charts are generated for each resource consumed by the building, and include:
*//Demand//: plotting recorded avg. power demand (solid line, denoted according to resource type in the legend) against Pulse's baseline demand (dashed line), computed using Pulse's proprietary (read: black box) analytics; in the legend, this reads as //Baseline 110301 to 120301//, indicating that the baseline was computed using recorded demand data from March 1, 2011 to March 1, 2012.
**A chart might have several baselines, computed using different date ranges.
**//Outdoor Air Temp// (in ࣡n also be plotted, though this is not included by default; its axis appears on the right of the time-series chart.
**The exact naming conventions for these variables are not consistent between charts.
*//CUSUM// (cumulative sum): these plot the cumulative savings in demand, computed using the differential from recored demand to the baseline.
**Typically these are plotted against a target or expected savings (y = mx + b) slope.
*//Control Charts//: these charts plot the //Differential from Baseline// and/or //Differential from Baseline (24h average)// against a fixed y-axis threshold (e.g. //Threshold @ 15 kW//, in which analysts are interested in periods where the differential between demand and baseline exceeds 15 kW).
**There does not appear to be a threshold chart for all building-resource pairs.
**Oddly, some of these charts include options to show demand and baseline time-series curves on the same plot, despite obvious the difference in scale, thereby plotting original and derived values on the same chart. Apparently this is a workaround due to the limitation of having only one chart visible at a time in the //Management// tab, while still taking advantage of the mouse-over and live updating of values. An example of this is a chart that shows demand data for the entire UBC campus on the same chart as demand data for UBC's comprising zones.
For each time-series plot, a mouse-over vertical bar moves along the x-axis, and the corresponding values for each curve are updated in the legend next to their labels.
//Messages// are also shown in the time-series plot, represented by symbolic labels attached to points on a time-series curve, or to points along the x-axis; in other words, messages can be associated with a particular dependent variable at a point in time, or to the a particular point in time itself. These messages include connectivity issues 'C', in which an interruption or problem involving the dataloggers that collect demand data. User messages ('U') associated with a dependent measure are also shown, often indicating important events relevant to the interpretation of the chart (such as the beginning / end of baseline computations; note that these must be annotated manually, and must be associated with a dependent measure). Known anomalies are also annotated using User messages, such as the activation of demand-intensive equipment in campus lab buildings (such as in the ~Chem-Physics building). Other anomalies include date shifting statutory holidays (Easter, Victoria Day, etc.), since Pulse's baseline computation does not take these into account. Building tune-up activity or //Energy Conservation Measures// messages ('E') are associated with the building at a point in time, and thus appear along the x-axis. //Threshold// messages ('T') are displayed whenever the dependent values trigger pre-specified threshold logic (duration and severity). Messages are revealed by mouse-over, and are listed in a table hidden in the //Messages// tab in the top right of the chart. //User// or //Energy Conservation Measure// Messages can be added/edited/deleted in the right click menu or in the //Messages// tab.
''Authoring a Management Chart'': (Note 1: many of the existing charts were generated by Pulse's JC, who was on an "analyst on loan" to UBC for a period of time at the beginning of the ~UBC-Pulse partnership. JC is responsible for the good chart naming convention. Note 2: chart authoring is not permissible for 'Viewer' users.)
These charts are generated in the //Configuration// tab, and involve selecting //Point// data and (optionally) building-specific //Baseline// and building-agnostic weather data. //Points// are either (1) dependent measures associated with a building, either raw demand values or derived values such as CUSUM or differential values; or (2) building-agnostic fixed threshold values (for comparing to differentials) or target slopes (for comparing to ~CUSUMs). For charts with building-specific points and baselines, the building must be manually selected for each. This is tedious because the building is not chosen at the outset of the chart-building process, and a single chart containing 3 or 4 time-series curves would entail finding and selecting the building from a list 3-4 times. Recommendations based on previously selected buildings would assist this process. It thus appears possible to compare one building's point to another building's baseline.
There are (apparently) no colour / line dashing conventions in the chart authoring process. It appears as though baselines are always dashed. LZ has slight colour-blindness, so some automatically-selected colour mappings and combinations are problematic.
Configuring a new baseline can take a considerable amount of computing time (hours? more? no expected time feedback is given). Thus configuring a chart can be a lengthy process: (1) configure building-specific baseline, (2) compute baseline (and wait), (3) configure building-specific points / derive CUSUM or differentials, (4) configure building-agnostic points, such as weather, fixed threshold values, or target savings curves.
While tedious, charts comparing points to baselines are the norm. However, some colleagues at UBC don't trust Pulse's black-box baseline computation, and merely want to compare two points, such as this year's demand to last year's demand, or to at least have the option of flexibly alternating between comparing against the baseline and comparing against last year's demand. If the variables considered in Pulse's baseline were made transparent, trust in its computation would be higher.
The time-scale can be altered from within the chart viewer, though in the chart configuration there is no guidance on how to select an initial time scale, particularly for generating CUSUM charts: what scale is meaningful? LZ relies on consultants for this information.
''Alerts'': //Energy Manager//'s features pertaining to //Alerts// are divided between the //Home// tab (listing alerts by building and links to charts) and the //Configuration// tab (time-series charts of reported values vs. thresholds, configuring threshold and alerts, listing thresholds, listing threshold events). There are two time-scales to be considered when it comes to alerts: instantaneous real-time events and historical semi-annually or quarterly trends. LZ does not want to be alerted with real-time instantaneous alerts sent to her inbox. Instead, she is interested in threshold trends that are reported on a semi-annual or quarterly basis. For instance, lab buildings on campus can be erratic and instantaneous alerts might be triggered often, especially if duration and severity thresholds are improperly configured (e.g. the Chem-Physics researchers turn on a powerful experimental laser device). Of greater interest are buildings that exhibit a trend that varies from projected performance; LZ wants to be alerted about these.
Alert setup is messy and distributed across the //Energy Manager// interface, and a considerable amount of alert logic must be considered in their configuration (recall [[Munroe, Shneiderman, Plaisant, et al CHI '13: "The Challenges of Specifying Intervals and Absences in Temporal Queries"|http://www.cs.umd.edu/localphp/hcil/tech-reports-search.php?number=2012-30]]).
''Reporting'': in the reporting tab, there is also a list of charts, some associated with single buildings, others with multiple buildings. These are distinct from real-time //Management// charts.
*The charts listed in the Reporting tab do not follow a consistent naming convention, and thus are not easily searchable by name, nor are they associated with buildings, resource types, or authors; they are displayed as a flat list, or optionally according to chart type: //Basic, Billing, Cumulative Savings, Historical Column, Load Duration Curve, Scatter, Top 25 - Peak Values, Weather Normalization//.
*There are different chart types beyond time-series line charts. There are hybrid bar/line charts, grouped/stacked bar charts, and scatter plots. Sometimes time is not considered and single annual values are compared across buildings or resource types.
*There is a single scatter plot in the portfolio, one of average monthly temperature (ࢥtween Apr 2012 and Mar 2013 vs. average power (kW). An optional y = 63.56 ⠰.1336 regression line is also included.
''Re: Energy Savings Potential'': For visualizing energy savings potential (for entire portfolio / a building of interest) LZ: might we just extrapolate from the CUSUM?
*Q: Are energy managers motivated more by projected savings or projected loss? By failing to meet targets, by meeting targets, or by exceeding targets? A cost of doing nothing vs. a savings for taking action. What is more motivating will dictate the type of information shown.
''Re: Pulse's Competition'': LZ mentioned a few competitors to Pulse's //Energy Manager//:
*The company that manufactures UBC's //Ion// energy meters, [[Schneider Electric|http://www.schneider-electric.com/]] does produce their own EMIS systems, such as their [[PowerLogic|http://www.schneider-electric.com/products/ww/en/5100-software/5115-energy-management-software/2251-powerlogic-ion-eem-395/?APPLICATION=1024]] EMIS system. LZ says that this slow is far too slow w.r.t. data import/export. [[ION EEM|http://www.powerlogic.com/product.cfm/c_id/2/sc_id/15/p_id/28]]
>//PowerLogic ION EEM enterprise energy management software exceeds the traditional boundaries of energy management and power operations software by uniting business and energy strategies across your entire enterprise while performing wide-area analysis of events and conditions.//
>//It is a unifying application that complements and extends the benefits of existing energy-related data resources. These can include power monitoring and control systems, metering systems, substation automation and SCADA systems, EMS systems, building and process automation systems, utility billing systems, weather services, spot-market energy pricing feeds, and enterprise business applications.//
*[[Energent|http://www.energent.com/]]'s EMIS [[dashboard|http://www.energent.com/technology/dashboard]] is highly attractive since it supports the creation of role-based custom dashboards, in contrast to //Energy Manager//'s single-chart focus and the distributed interfaces for //Management//, //Reporting//, and //Configuration//.
>//The Energent Dashboard is an easy-to-use and fully-customizable window into your energy use. It provides a snapshot of your energy performance and access to essential analysis tools and detailed reports.//
*[[NRCAN|http://www.nrcan.gc.ca/home]]'s freely available [[RETScreen|http://www.retscreen.net/ang/home.php]]. LZ describes this product as having logical and transparent benchmarking/baselining considering multiple variables, such as multivariate weather data. (Note: there was only one weekly average energy demand vs. outdoor temperature scatterplot in the amongst the reports listed in the //Reporting// tab.)
>//~RETScreen 4 is an Excel-based clean energy project analysis software tool that helps decision makers quickly and inexpensively determine the technical and financial viability of potential renewable energy, energy efficiency and cogeneration projects.//
!!Document Notes
LZ sent along contacts for UBC BMS and SES consulting, as well as a number of documents following the interview:
!!!Memorandum: User feedback on Pulse EMIS (LZ > Pulse's JC, Jan 31 2012)
A "wish-list" table of features "that would improve usability", with priority rankings (1-4, 1 being highest), with subcategories for Architecture, Charts, Reports, Analysis, and Process:
*Architeture:
**navigating between charts, reports, and points (chart configuration) is tedious, as is navigating long lists of buildings (2);
**standard nomenclature / naming conventions for charts, reports, points (automatically generated as opposed to user-defined names), use of building, time, resource/element type, point type metadata (3);
**ability to recall / restore recently viewed items/charts, recent settings, filter params (3)
**a desire for custom user-specific building portfolios and/or building tags (zones, operator names, phases) (4);
*Charts:
**//Ability to compare two different time periods either in two separate charts stacked on the same screen, or with two curves displayed in the same chart.// (e.g. month-to-month, week-to-week comparison (1);
**fixed y-axis scale (not autoscaling for each time period) (1);
**ability to choose custom date range in management charts beyond pre-specified default options (2);
**OAT (outside air temperature) line to be above the energy chart / separate chart, not crossing the energy curves (2);
**multiple and different chart types on a single screen (not just time-based), a holistic portfolio / dashboard experience for a building, perhaps user-defined or role-based (2);
**standard colour / dashing conventions - not to be configured or different for each chart (3);
**universal unit selection / location where this information is always shown (3);
**control charts (actual - baseline differential) to be viewed side-by-side with raw consumption time-series values (4);
**stacked bar charts for buildings with multiple resources consumed (4);
*Reports:
**report templates with minimal setup (select building and date range); saving parameters rather than reports themselves (2);
*Analysis:
**weather normalization, comparing weather-normalized time-series data (1*)
**statistics on baseline / typical curve accuracy, a means to classify a building as regular or erratic or irregular, assist in setting appropriate thresholds and MT&R (Monitoring, Targeting, and Reporting) pratcices for that building, track changes in the accuracy of a building's typical curve (is the building becoming more predictable?), compare training curves from before continuous optimization to after continuous optimization (or any custom time range), summarize all buildings according to predictability and changes in predictability (1);
**separate out baseload and weather-dependent portions from data, or breakdown into stacked bar chart (2);
**automatic weather-normalized portfolio benchmarking (BEPI: Building Energy Performance Index, "//calculated by dividing a space's energy use over a calendar year by its floor space//", or EUI: Energy Use Index?) (3);
**understand baseline accuracy as a factor of time (when is baseline certainty high/low?)
**ability to track baseload increases (4);
**toggle energy savings vs. cost savings (4);
*Process:
**guidance for setting appropriate thresholds (time window, percent differential severity), impact on severity and frequency of alerts; "//Could a more informed decision be made by analyzing the regularity of the building and other patterns in the buildingॲation?//" (1);
**guidance on who is responsible for creating reporting points, training baseline curves, setting thresholds: pulse, client, consultant (2);
**awareness of baseline curve calculation completion (is it ready to use in setting up threshold alerts?) (3);
**follow-up on threshold alerts - were they acknowledged? currently this is not reported in the //Management// tab where threshold alerts are listed, but in the //Configuration// tab (3);
!!!Daily Consumption Report Design Mockups (c2010)
Pulse hired a graphic designer / user interface designer to create mockup daily consumption reports; 5 of which were sent by LZ.
Overall: highly saturated colours, compact/dense displays.
*p1: an overview of daily consumption values for multiple buildings, arranged in a table, similar to the ranking table on the //Home// tab of //Energy Manager//.
*p2: a daily report for a single building with multiple resources: 24h demand and temperature line chart, monthly target comparison (calendar chart, demand bar chart)
*p3: a daily report for a single building with multiple resources (same as p2, alternative colour scheme)
*p4: a monthly report for a single building with multiple resources: monthly line chart w/ temperature, yearly target comparison (calendar chart, bar/line chart)
*p5: an annual report for a single building with multiple resources: bar/line chart, yearly target comparison (scatter plot comparison to baseline)
!!!Protocol for Monitoring, Targeting and Reporting (MTR) system (LZ > PB, UBC BMS Manager, May 2013)
A memo sent to the manager of UBC's building operations, defining the MTR system, Continuous Optimization. Discusses MTR SOP and revisions to the SOP: tasks, workfows, and responsibilities.
|Matrix of key MTR tasks|c
|! Freq. |!Reporting/Monitoring Task |!Tools |!Staff |!Est. Effort / cycle |!Hrs/yr. |!Audience |!Venue |
|Annual |C.Op. savings handoff (post-handoff) |Excel |ECM |4 days |28 |BOLT |Presentation |
|Semi-annual |CUSUM check of irregular buildings - heating and cooling seasons |Pulse, Excel |ECA |5 days |70 |Managers, HVAC, BMS, Energy |Meeting |
|Monthly |CUSUM check of regular buildings |Pulse, Excel |ECA |4 days |336 |Managers, HVAC, BMS, Energy |Meeting |
|Weekly |Check on worse buildings |Pulse (via login) |BMS Ops. |1 day / week |322 |N/A |N/A |
|Daily |Check threshold alerts |Pulse email alerts |BMS Ops. |1 hr / day (avg) |230 |BC Hydro requirement |Email |
From 07-09-13 email:
>//For the laboratory buildings in Phase 1 of Continuous Optimization, a semi-annual check is proposed since much variability is expected due to irregular operating patterns and research activities. For classroom, office and library buildings in Phase 2 and 3, a quarterly check is proposed since they behave regularly with well-defined patterns.//
>
>//Heating season operation will be checked in January/February to allow for diagnosis and correction before the end of the season. Cooling season operation will be checked in July. Quarterly checks will be added in April and October.//
Notes:
*ECM = Energy Conservation Manager? (LZ); ECA = Energy Conservation Analyst; BOLT = building operations leadership team; C.Op. = Continuous Optimization; BMS = UBC Building Management Systems; HVAC = Heating, Venting, and Air Conditioning;
*Pulse vs. Pulse (via login)?
*Continuous Optimization has been applied to only 2 pilot buildings on campus since 2011
*MTR is being conducted for over 80 buildings on campus
*2 types of anomalies: short-term alert thresholds and gradual drift from target performance; latter is different for regular/irregular buildings
*workflow: BMS Ops and ECA analyze and delegate, BMS technicians and tradespeople diagnose and subsequently re-program and repair
!!!Error Analysis of Baseline Curves
An Excel spreadsheet comparing electricity and steam baselines for 17 campus buildings (some having multiple baselines spanning different time windows). For each baseline, lists the type and version, the number of exclusions and the date of the last exclusion (if applicable); for electricity baselines, the interval error and confidence interval; for steam baselines, the R^^2^^ Error and base temperature; comments for changes to building programming/infrastructure are also included.
Notes:
*Baselines range from 5 to 42 months in length
*Re: Baseline Type and Ver: What is nomenclature? BL (#) / LR
*Exclusions: what are these? Why report the date of the last exclusion? some buildings have no exclusions, while others have as many as 18.
*Interval error ranges between 0.7 and 16.7, confidence intervals between 1.3 and 28.9
*All R^^2^^ Error > .0.92
*Base temperature ranges between 12 and 19ᡍonitoring, Targeting and Reporting (MTR) Standard Operating Procedure (SOP) (Sept. 28 2011)
A 16p draft SOP document prepared by SES consulting + 20p of appendices, includes a public statement, MTR breakdown, implementation procedures, SOP revision control, and ECM (energy conservation measures) definitions.
Appendices contain Pulse Energy Support documentation (an abridged version of what is found in the online help), targeted at UBC BMS Ops personnel
Notes:
*Much of the SOP is captured in the above notes and protocol for MTR memo (above)
MT, M.Eng, energy specialist at Surrey School District (SSD).
!!MB notes:
MT has worked with Pulse Energy Manager (EM) for 2 years, in a managerial role at SSD
He refers to two flavours of analysis workflows, one at the "macro" level (comparing portfolio rankings) and one at the "micro level" (occupancy profiles of schools, comparing day-to-day, actual-to-baseline). Currently, EM is primarily used to export data for analysis in Excel.
''Portfolio'': There are currently 31 of ~130 SSD schools in EM, as part of the BCH C.Op and ~EnerTracker programs. Eventually, all 130 schools should be in EM, (C.Op or no C.Op). There are two separate EM accounts: one for electricity through BCH, and one through Fortis ~EnerTracker for natural gas. Switching between the two resource-based views of the portfolio require logging out and back in with the other account. The ~EnerTracker has the new EM home tab, while the BCH C.Op SSD does not. There are 4 SSD districts, though it does not appear that this hierarchy of districts and schools is reflected in EM. Several of the schools are submetered (notably Adams Road and Woodward Hill). One of the largest consumers of energy is the SSD District Education building (the building where MT works), which is not submetered.
''Macro-level'': One of MT's workflows is to rank schools in terms of their energy intensity within the SSD portfolio. This is a necessary element of his quarterly reporting on the status of the BC Hydro (BCH) C.Op program and the energy usage of the SSD portfolio. MT compares the ranking of one quarter to the previous. This rank comparison is normalized based on area in square footage (though area can be dynamic for partial non-school-hour rentals). The ranking is also normalized for weather based on the number of heating degree days. He wants to spot anomalous changes in position in rankings, and identify high-level trends, such as schools that are routinely near the top or bottom of the ranking.
Another macro-level comparison is between absolute and normalized savings.
''Micro-level'': Refers to analysis at the single-school level, aka troubleshooting. MT's goal is to ensure that the buildings are automated such that the appropriate amount of energy is consumed given the occupancy of the buildings. He needs to correctly detect operating hours and three occupancy profiles. There are marked differences in occupancy between school hours (8:00-16:00), after-school (evening) hours (16:00-22:30), and night hours (22:30-8:00); the latter of which should, in theory, be marked by zero occupancy (building off). After-school hours are variable: custodial staff, extra-curricular events, and event rentals. Event rentals involve partial use of a building, and bring in money to SSD, though the cost of energy required for this partial occupancy may be greater than the rental fee. He also compares weekdays, weekends, holidays, for example comparing all Mondays in January, or Mondays to Tuesdays. Comparing to last years' data is also done, and he makes sure that the correct dates line up (considering holidays). Ultimately, his overall macro-level ranking is based on the quarter-to-quarter performance, and in order to understand changes in that ranking, he needs to drill down (get details on demand): a breakdown from a year-to-year comparison, comparison by quarter, month, by weekday, and by time of day.
MT also wants to identify peak demand days and determine their spread, or determine whether they are clustered; he needs to see the peak demand which sufficient context, what happens before and after the peak.
''Current methods'': In Excel spreadsheets, he organizes energy intensity data for all 130 SSD schools; these spreadsheets make use of custom macros, a customized colour coding scheme based on periods of the day of interest to his reporting: school hours, evening hours (custodial staff, extra-curricular events, building rentals), and night hours. Currently it is difficult to spot anomalies based due to the number of buildings in the portfolio and the time interval colour-coding scheme he is using.
''Alerts'': EM's alert feature is currently not in use, as he expects it would be difficult to keep track of alerts for all 30 schools currently in the system, let alone all 130 buildings should they eventually be reported within EM. He cites the paradox of choice, in the absence of recommended default thresholds, perhaps based on recorded baselines, the choices of potential threshold alerts for the 30 buildings in EM is overwhelming.
''EM Performance Ranking'': the EM performance ranking table is likely to be similar to the ranking that MT does in Excel, for a quarterly date interval, though he has not compared these rankings.
''Baseline'': MT would like to see more references to what the Pulse adaptive model / baseline curve is doing, how it is calculated, what its limitations are (representation of uncertainty, confidence intervals, perhaps); he wants to know what temporal window was used to calculate the baseline, whether or not anomalies are omitted, whether it normalizes weather variables, or normalizes for other factors, such as occupancy.
''Other SSD users'': MT also oversees a volunteer program (BC Ambassador program), which enlists SSD students, typically high-school seniors to become familiar with EM and Dashboard report anomalies. MT likens this to the City of Chicago open source energy reporting initiative, which received VC funding for building retrofits and monitoring.
''Misc'': he recalls seeing Pulse's case study articles (find articles).
KT showed some mock-ups of EM redesign, including a design addressing the workflow of identifying creeping baseload. MT admitted needing more time to absorb this workflow, which is something that he's not currently performing.
!!KT's notes:
>//MB and I had a productive session with MT of the Surrey School District (SSD). Here is the summary of the notes I took in our session.//
>
>//I've attached a few pdf's that MT has asked we keep confidential.//
>
>//The district has 130 schools, with 31 in the system (C.Op and EnerTrk).//
>
>//They are doing some really interesting work with the BC Hydro Energy Ambassador Program, with the help of Harish. They're starting with a 5 school pilot where he is training students in those schools to use Energy Manager and the Dashboard to watch for problems, report, and troubleshoot. See the attached SD36_ECP2_Course 3.pdf for some of the training material.//
>
>//I think they could use all our products. Pulse Check would be a good tool for MT to help increase engagement at schools among teachers and principles and it would likely be a better tool for the student ambassadors to use with the dashboard.//
>
>//MT sees two distinct workflows, what he calls macro management and troubleshooting. We'd call macro management portfolio management. MT would like to spend more time doing macro management and less troubleshooting.//
>
>//For macro management MT uses Energy Manager to pull data and do Excel magic. Some of what he does in Excel he could do in the software though his spreadsheets include data about the schools that aren't in our system. The data could be in our system e.g. operating hours and occupancy but in Excel he adds color coding to cells and he needs to analyze day, night, and evening demand for all the schools. He essentially ranks the schools by intensity and total consumption and focuses on the worst performers until the ranking changes. Most schools have DDC systems so he can make remote changes.//
>
>//For troubleshooting, MT uses EM charts and reports. See the attached ...Bi-weekly check protocol.pdf for some of his workflows.//
>
>//''Miscellaneous''://
>
>//''Alerts'': doesn't have time to manually configure alerts for 30 schools. Wants them auto-configured and he can turn them off if needed. Only wants to receive an e-mail when it is important.//
>//''Baselines'': needs to know what the baseline training period is and whether crazy time periods have been excluded.//
>
>//''Demand comparison'': Would like to compare all Mondays against each other and to an average weekday over the same period. Also compare stat holidays to regular weekends.//
>
>//''Rankings'': would like summary of higher trends, # of schools improved, gotten worse.//
>//He doesn't use the Portfolio page rankings: colors are too confusing, not sure what the savings numbers mean.//
>
>//''Peak demand'': wants to see list of 25 occurrences and on the chart.//
>
>//''Weather normalization'': has built a chart (in Excel) that shows normalized vs non-normalized consumption side-by-side.//
>
>//''Load duration'': would be better if it was points, doesn't understand why it's a curve.//
>
>//''Start-up time'': would like this to be auto-detected and shown somehow.//
>
>//''Consumption by time of day'': one of his key workflows is comparing consumption at day, night, and in the evening so he can see how each school is performing in each period.//
>
>//''Comparisons'': he needs to see charts for all his schools at once so he can compare them. He painstakingly loads the home page for each school, copies their consumption, then puts together a big jpg image that he prints out! //
>
>//MT was overall positive about our software and our company. A high energy guy, passionate about what he does, and I enjoyed the visit.//
>
>//Cheers,//
>//KT//
!!Document Notes
1.Email exchange b/w MT and SL (Aug '13)
*MT's review of EM and its functions associated w/ the ~EnerTracker program. comments:
**a need for baseline transparency, whenever saving amounts are noted; "comparing what-to-what"? What baseline time window? Some savings may not be relevant
**better colour contrast for temperature comparison
**alerts may be useful, but more feedback (guidance?) is needed
2.Email exchange b/w MT and JC (Sept. '13), (quoting earlier exchange with HR)
*lists a need to support default comparisons:
**monthly consumption patterns over a year for a single school (see attached mockup: Bear Creek 12.13 Overview)
**sample of 4 different [weeks, days] across the year, weekly and daily (see attached mockup: Bear Creek 12.13 Weekly and Daily)
*re: alerts - a need to set up an alert for "unusual activity" at night, a surge or spike - needs guidance on how to set that up
*quoting an earlier exchange with HR, he describes 3 occupancy intervals (school hours / day, after-school / evening, and night) and a need to see consumption broken down into these 3 occupancy intervals; see mockup of Earl Marriott Secondary Combined Consumption graph in which he alters the bar chart of electricity consumption to a stacked bar chart that reflects consumption broken down into those 3 intervals (note: the colour/position channels would be overloaded, as combined consumption is already a stacked bar chart representing different resource types - SSD has electricity and gas reporting separated between two EM accounts)
3. Mockups by MT
*Bear Creek 12.13 Overview: combined consumption small multiples for 12 month period, 1 month per chart (constructed with screen captures from EM)
*Bear Creek 12.13 Weekly and Daily: 2 columns of combined consumption profiles: 4 weekly (left) and 4 daily (right) consumption profiles from a sample of 4 different weeks / days of the year (constructed with screen captures from EM)
*images002/003 - two monthly combined consumption profiles for Earl Marriott Secondary in January, one unaltered (screen capture from EM), and one edited that shows January 9th's consumption as a stacked bar indicating day (original yellow), evening (red), and night (black), with percentage figures overlaid
4. Energy Ambassador PEMS Introduction
*a short 7p course authored by MT, an introduction to EM
**provides guidance on making good use of Pulse EM: what to look for, using the portfolio performance summary and consumption overview, how to export charts, create alerts, create custom points and baselines, generate reports (an abridged / paraphrased versions of EM's existing documentation)
5. Energy Ambassador Program protocol documentation (~PDFs)
*A 6p protocol, with images, for SSD students using EM and Pulse Dashboard to report on anomalies and trends, a bi-weekly check protocol, broken down into analysis protocol and result reporting protocol
**Macro-level (all schools) - EM dashboard, counting how many schools are doing better/worse over past week
**Micro-level (your school) - log consumption pattern, peaks, after-school-hour activity, weekend activity, alert level for a week
***if an abnormality is spotted (e.g. spike, surge, increase/decrease), log details of anomaly
***compare demand to average demand over past week, log peak demand
meeting w/ MT, SSD energy specialist - Dec 09, 2014
!!!Macro-level analysis
*MT manually exports annual consumption data from EM for 40+ schools, imports into Excel to examine the year-over-year % changes in consumption, juxtaposed with a list of ECMs or other important events for each building; based on information in this table, MT creates a ranking of the "worst" performing sites based on overall % change, detection of a trend toward more consumption, an anomalous peak event, a combination of the above, and/or if behaviour unexplained by ECMs / known events.
*Multiple site comparison is done at an aggregate year-over-year time scale (not quarterly or monthly), a single consumption value per site per year is considered.
*It would be useful to have one table that reports the peak value for every site in the portfolio.
*MT says that looking at data for a single site doesn't tell you much, even with baselines; it's helpful to look at multiple sites, normalized by square footage (intensity).
*There have been occasions when MT started a reporting workflow at a macro level, but discovered an anomaly that is worth following up on at the micro-level.
*If MT were to begin a lighting retrofit project at a single site, MT would want to tell [EMU] the start and end dates and have these highlighted in any charts containing that time span.
*''Q'': what are your top priorities? ''MT'': this depends on time span: MT finds the 10 worst performing sites using his ranking scheme (described above), and comes up with an actionable measure for each of these, as a way to prioritize retrofit projects.
*An example of an anomaly: Kirkbride Elementary School had an anomalous increase in consumption this past year, with no explanation for the gain.
*One of the worst performers in MT's portfolio is the only site with artificial cooling, so bad performance is obvious and expected when you have knowledge of the building; for this building, MT would compare September (a month that is typically without cooling or heating) to June (students are still there, so occupancy is comparable to September, but cooling is required due to higher temperatures).
*''Q'': normalization by temperature or HDDs/CDDs or by temperature? ''MT'': normalization by heating degree days important for natural gas, not for electricity.
*MT has to make the case for upgrades: energy efficiency is not enough to justify a retrofit, such as replacing a boiler; at this point, all the obvious retrofits in the portfolio have already been implemented; suggestions for less obvious retrofits come from his rankings; the protocol is then to secure AFG or COp funding; some retrofit projects are initiated by the facilities department; MT also makes recommendations to the facilities team, however the facilities managers' main priority is safety and comfort at individual sites.
*MT wouldn't mind if the software asked him about possible anomalies that it detects, asking him to explain anomalies with a message (an example explanation from his portfolio could be a partial building rental period).
!!!Micro-level analysis
*MT currently doesn't set up alerts for sites in portfolio via EM; MT could set up alerts on a site-by-site basis, but would prefer to start with a portfolio-wide systematic alert that could be tweaked on a site-by-site basis based on the unique or specific aspects of each site; MT would prefer to see alerts in a table, an overview of alerts: MT prefers to "think in batches"
*MT would appreciate an algorithm to detect when and how a building is starting up and shutting down: automatic detection of distinct time periods, not just on/off hours (in his case, MT has three distinct periods: school hours, evening, and night); automatic detection of time periods should not preclude manual tweaking of these periods afterward; custom hours: initially MT would specify the same opening and closing hours for each site, which should be customizable for each site; the software should detect three periods and provide ability to tweak these time periods.
*MT wants to determine if fluctuations within time periods are human-related: during school hours, not much can be done since most fluctuations are human-related; the evening period is variable, and during the night all sites should be shut down in theory: this is when you detect things like fans failing, by examining nighttime base loads.
*MT doesn't do a daily monitoring using EM, but uses it once every week or two weeks: MT extracts data from each building to Excel, examines and compares time-over-time, looks for anomalies, especially at night when the building should ideally be quiet and have little fluctuation, MT may be able to detect problems such as failures in lighting automation or fan failures,
*MT doesn't want a continuous daily stream of alerts from EM: MT cites an occasion where a large fan in a building failed, and it took 3 months to identify and resolve; a 1-2 day anomaly is not important, but a week-long anomaly is worth the attention and MT should be made aware of it.
*His main concern is missing an expensive event (such as the fan failure example), something that was obvious in hindsight but was not brought to his attention.
!!!Miscellaneous
*re: the energy ambassadors program: no longer active, pilot period is now over; biweekly reports using consistent questionnaire spreadsheets; the participants lacked experience (secondary school students), minor fluctuations flagged by students were not worth following-up on.
!!!Demo
*re: multiple site load profile (site overlay view, not tile view) for October 2014, with a 4hr granularity: found an example of one site that exhibits higher load profile on one weekend in October 2014: what is the cause? MT would want to leave a message here (annotation drag + drop is currently not working).
*re: multiple site load profile (tile view): this view option was well received (similar to his EM mockups); MT expects to be able to zoom in within one tile, and that this wouldn't affect the zoom levels of the other tiles; upon further consideration, you could just as well link the zooming with the other tiles, so when you zoom in one tile, all other tiles are similarly zoomed; alternatively, if he zoomed in on one tile, he probably doesn't care about the other tiles any more (he's no longer comparing, but examining a single site at a finer time scale), it could expand the tile where MT zoomed to fill the screen.
*Missing data should be flagged in each view; currently when exported from EM, they are represented as zeros, which was a problem for MT when extracting and tallying up aggregate data in Excel, the results didn't match the utility bill; since then, MT created a custom macro to detect intervals of missing data when exported from Pulse; this experience reduced trust in Pulse's reporting and exporting functionality.
*The time breakdown load profile view was well-received (MT looked at each Friday in October 2014 for a single site).
*MT didn't like beginning with load profiles of demand; typically MT begins his analysis by looking at consumption bars; perhaps if MT found some anomaly when looking at historical comparisons of consumption, MT would then investigate the finer-scale demand data.
*re: historical comparison view: in addition to comparing to a historical period, MT would also want to compare to an average value over the whole year (but not a adaptive baseline?).
*re: the heatmap: the heatmap colour scale is too subtle (light red to dark red; buildings in EMU are too stable); the heatmap presents two dimensions of space and time, which presents more choices to the user; MT says this is OK for an advanced user, but not for a new user.
*re: the boxplots: MT is familiar with them and understands their purpose as being complementary to the heatmap, though he hasn't seen them used before in energy analysis.
*re: ranking visualization: MT sees value in this, especially since he currently performs multiple annual rankings in Excel, though the time intervals would be annual, not quarterly or monthly.
RS - LTC - Dec 09 2014
!!!Background
*Job title: operation manager for LTC (S㓠properties).
*Receives daily reports from EM, looks at report daily, doesn't login to EM.
*Top 3 responsibilities: mall maintenance, comfort and safety, projects.
*Energy related tasks: examining outside air temperature, examining trend logs for mall temperatures, ensuring that energy schedule isn't coming on too early or staying on tool late; max 2hrs / week.
*Retrofit projects for the mall were identified in an initial audit preformed when the building was purchased by S㓮
*They haven't yet implemented an energy savings retrofit yet; they are currently implementing a lighting retrofit.
*''Q'': do you produce reports? ''RS'': Yes, RS shares the EM report with [supervisor], chief operating engineer (the latter doesn't get too involved, but is aware that a report exists).
*Building maintenance is performed by a 3rd party contractor, who are kept informed about schedules, shutdowns and startups.
*''Q'': what software is used to monitor heating/cooling data, for gas and electricity? NUS software; overlaps with Pulse; NUS is sent all the invoices and they prepare a report; separating out in terms of heating and cooling. NUS reports are not very accurate, so charts are ignored; they are delayed by a month. NUS provides raw data, not CUSUM: someone else does this transformation.
*At RS's former job with the Justice Institute, the software generated CUSUMs.
*''Q'': what are the savings goals? What are the measures? ''RS'': not determined yet. Energy intensity is an important metric; occupancy can be approximated to determine energy consumption normalized by occupancy.
*RS has had no time to explore EM, so he relies on daily reports and responds to requests.
*Energy-related tasks takes up a couple hours a week.
!!!EMU Demo
*"Plan-Optimize-Verify" cycle is not familiar, but "makes sense".
*RS is familiar with load profiles; he would look for anomalies, deviations from a baseline (baseline data not yet available / presented in EMU).
*The S㓠Properties portfolio: LTC food court, LTC boilers (building), BTC (another building).
*RS imagines looking at EMU with a member of his team.
*Thoughts on load profile interface: it's not overwhelming, not overly populated.
*RS likes the tooltips in the visualizations.
*Multiple sites: RS probably wouldn't look at multiple sites in so much detail (though he doesn't have multiple sites).
*On time-over-time comparisons: a year-over-year comparison is handy.
*Weather information is necessary for time-over-time comparison.
*HDDs/CDDs puts energy consumption into perspective better than looking at a chart (?)
*There is no baseline data currently in EMU. ''Q'': what other variables should be considered into the baseline? weather (OAT?) has greatest impact.
*Alerts are not yet arranged; RS wouldn't want nuisance alerts.
*Week-over-week analysis: "that's cool".
*Date selection confusion: historical date range is confused with current date range.
*Weekday-over-weekday historical comparison mode is not currently working; RS hasn't done this type of analysis before at his previous job (at the Justice Institute).
*At a mall such as the LTC, weekday to weekend comparison may be interesting (higher occupancy on weekends). RS would weekends due to fluctuations in usage at the LTC.
*Focus + Context time selection in consumption is not obvious.
*Trust in baselines would come with experience using EMU.
*re: Energy vs. temperature: very sparse data (only ~5-6 points for any given date range: bug?)
*re: Heatmaps: RS hasn't seen this type of chart before, but it makes sense - boilers are turned off in summer months.
*EMU provides a sense of greater immediacy and day-old data, relative to NUS's month-old data.
*RS usually examines kWh consumption rather than cost or demand.
*re: the usage scenario of Fair Hotels, filtering by tags (note: tags not likely to be of interest if user has a single building, not a portfolio)
*The word "tag" is unfamiliar, "label" might be more familiar; RS first thought of "tagline" as used in marketing.
*Date selection: RS still has tendency to use historical date selection rather than current date selection.
*''Q'': Demand-response? There isn't a lot of wiggle room in terms of turning off lights / heating: comfort and safety are more appropriate than efficiency.
*''Q'': Mobile EMU? Could be valuable, but there is no precedent for this.
Meeting w/ [[CG, SES Energy Efficiency Engineer|http://sesconsulting.com/about/people/]]:
>//CG has worked in energy efficiency for two years specializing in building optimization. Prior to working with SES, he worked as a systems engineer focused on the performance of fuel cell engines. CG has a B.A.Sc. with distinction in Mechanical Engineering from UBC, which included two semesters at the University of Western Australia. CG work at SES includes investigation and implementation of advanced measures to improve building energy performance and occupant comfort.//
>
>- bio from [[SES|http://sesconsulting.com/about/people/]] website.
CG is responsible for performing the early-stage investigation of the energy consumption of buildings; for clients such as UBC, these buildings are marked for BC Hydro's Continuous Optimization program. In other cases, a client with a portfolio of multiple buildings may approach SES to determine how to prioritize future building optimization efforts. In the former case, it is CG's job to prioritize these optimizations for a single building. In the latter case, these prioritized recommendations for optimizations are both across and within buildings. To do this, CG uses Pulse as a preliminary EIMS (Energy Information Management System); CG also has experience working with Energent, a competing EIMS tool, though Pulse is preferred for its openness and usability, and for the apparent avaiability of data. Typical/baseline curves generated by Pulse are seen as transparent in that it is clear as to which date ranges were used in their construction.
There are several phases to this work, beginning with an investigation phase of current energy demand and consumption, followed by a presentation of recommended optimizations to the client; after changes have been implemented, Pulse is used to generate typical baseline curves and establish thresholds and alerts; subsequently a coaching and hand-off phases occurs in which the client takes over analysis of their optimized buildings. CG is responsible for the early investigation phase, while SES' NV is responsible for the later coaching and hand-off phases.
Initially, a building's energy consumption will be examined using Energy Manager's management charts at a small time scale, usually at the level of a day. This process is repeated a few times to get a sense of a building's typical weekday and weekend load profiles for the four seasons of the year. For most buildings, a load profile will reflect its control settings and will be easily apparent after viewing a few charts. For erratic buildings, it make take more investigation and review of several days' worth of consumption data to understand their load profiles and anomalies or differences from their control settings. in some instances, a building will be compared to another building with a similar square footage to determine whether it is behaving erratically or normally and whether it is deviating from its control programming. Whether a building is "good" or "bad" is also captured by Pulse Energy Manager's Performance Ranking on the Home/Portfolio tab, however these labels are determined relative to the building's typical or baseline curve, which is yet to be computed at the stage when CG is examining building data. As such, CG does not use this information (or tab). Instead, CG is interested in generating a ranking of opportunities rather than a ranking of performance relative to Pulse's typical/baseline curve, and this ranking of energy savings opportunities may be with respect to a single building, or spanning several buildings in a portfolio.
To develop this ranking of energy savings opportunities, CG exports information from Pulse and imports it into Excel. For large projects with multiple buildings and/or multiple charts, loading these Excel files can be slow. With this data, detailed exploratory analysis is performed*, beginning with a template of analytics and custom charts. This analysis template is constantly being refined, however this takes a considerable amount of time and effort; it would be ideal if all of these charts could be generated in Pulse. CG speculates (and based on my interaction with UBC's Pulse portfolio, I agree) that many of these charts can be generated in Pulse, however authoring these charts and configuring points within Pulse is tedious and is less straightforward than generating these charts in Excel, which is seen as transparent for an experienced Excel user such as CG. The downside to these charts is that they do not allow for interactive exploration or tweaking of optimization parameters. Many of the charts are time-series charts, including scatter plots of Outdoor Air Temperature (OAT) to weekly energy consumption, multi-line time-series charts that aggregate daily consumption for the seven days of the week, separated by season, across 2 years of collected data. To understand day-to-day fluctuations, demand is often examined and compared at small time scales (e.g. comparing 2-4 pm on Monday afternoons, for instance). Ideally it would be possible to normalize OAT, other weather variables, building occupancy, building square footage, and time of day. Raw demand data as well as rate-of-change/slope and standard deviation data is examined, as well as average, minimum, and maximum curves; unusual peaks, troughs, and day-to-day differences are investigated. For multi-building comparisons, the data is normalized by building size and occupancy in order to determine how to prioritize C.Op recommendations, as the relative savings an implemented change will incur is not initially obvious without this normalization.
Once a building's load profile is determined, it is possible to SES to consider additional information about the building to understand where the energy is going, whether it be to HVAC, equipment, lighting, or to other uses. Ideally this could be estimated with a mechanical survey of the building, without having to physically inspect the building and its control setting; however, it is inevitable that at some point in the C.Op cycle, SES will perform site visits. Attribution of energy consumption to various sources within a building is not required to be completely accurate, but it does factor into SES' recommendations. This information is usually represented as a pie chart. Eventually with smart meters and related technology, it will become easier to accurately allocate energy consumption within a building.
Together, the source-attributed load profile and the normalized typical day-to-day demand charts inform the analysis of simulated changes in building control programming to selectively reduce demand. The resulting simulation data is used to project energy/cost savings, which is presented as a prioritized list to clients. It would save a great deal of time if this simulation and analysis of energy savings could be interactive within Pulse's Energy Manager, rather than computed and charted manually in Excel.
!!Document Notes
*Excel documents with detailed comparisons and charts
*Meter Analyzer Tool.xlsx (crashes Lenovo x86 ~ThinkPad)
**An incredibly large table (35K Rows, 120 columns), 15 minute interval data, electrical demand, hourly OAT, month, year, date, time, day of week, season, kW/㊪*demand histogram
**OAT histogram
**aggreagte data quality measures by month/year
**monthly aggregate consumption history by month/year (2011 only)
**peak monthly demand by month/year (2011 only)
**load factor by month/year (2011 only)
**OAT vs. demand with breakdown based on occupied / unoccupied times of day and by heating/cooling inflection points + scatterplots (all, binned average, binned stdev)
**breakdown of consumption into baseline energy, heating and cooling energy
**seasonal weekly profile table, avg. per time of day and season (table mostly full of zero values), 15 min intervals + (flat) line chart
**schedule line charts (spring, fall, summer, winter) (''MB'': what is this?)
**seasonaly weekly profile table, ST deviation table, 15 minute intervals + line chart
*Group 2 - BUILDING SUMMARY.xlsx: data and charts for 5 buildings:
**OAT histogram (time window 3 years: 2010-2012?)
**demand histogram (time window 3 years: 2010-2012?)
**peak monthly demand grouped bar chart for 2010-2012 (all months/years not reported for all 5 buildings)
**monthly consumption grouped bar chart for 2010-2012 (all months/years not reported for all 5 buildings)
**monthly load factor table for 2010-2012 (all months/years not reported for all 5 buildings) (''MB'': what is load factor?)
**OAT vs. demand scatterplot (actually, a line chart along x-axis, resulting in blob/hairball)
**peak demand during a week (min, avg, max)
**seasonal weekly (demand?) profile line chart (winter, spring, summer, fall): maximum
**seasonal weekly (demand?) profile line chart (winter, spring, summer, fall): average
**schedule line chart: winter, spring, summer, fall (''MB'': what is this?) - difficult to sepearate curves
**seasonal weekly profile st. dev line chart: (winter, spring, summer, fall)
**seasonal weekly profile st. dev (normalized) line chart: (winter, spring, summer, fall) (''MB'': normalized by HDD?)
Meeting w/ [[NV, SES Energy Efficiency Engineer|http://sesconsulting.com/about/people/]]:
>//NV has worked with SES Consulting for 4 years performing energy audits, Continuous Optimization studies, and carrying out energy project design and implementation. She received her BSc from the University of Illinois at ~Urbana-Champaign and her MASc from UBC, both in mechanical engineering. Her graduate research work involved developing and testing a method for enhancing efficiency in sawmills in order to reduce demand for raw material. NV specializes in DDC optimization and troubleshooting high energy use.//
>
>- bio from [[SES|http://sesconsulting.com/about/people/]] website.
!!Notes
NV handles the coaching and hand-off stage of the C.Op progam (whereas [[SES's CG|Pulse-SES-CG-13.08.06]] performs the early analysis and C.Op target building identification analysis). She has used Pulse for several years in many projects, including C.Op programs at UBC and SFU
NV has considerable experience using Energy Manager (EM) as well as well as other EIMS systems, namely [[NorthWrite's Energy Expert|http://www.northwrite.com/energyexpert.asp]] ~Real-Time Monitoring Software, which was used at SFU (also see ~NorthWrite Score Card email). ~NorthWrite sends a daily email digest reporting on the energy consumption behaviour an individual building. NV used Pulse EM for the C.Op program at UBC. She notes that EIMS engagement at SFU is lower than at UBC, for the buildings at SFU monitored in Pulse EM, there are no notes, alerts, baseline curves.
''Thoughts on Pulse EM'': The drop-down aggregated hierarchy of UBC buildings on the top left of the home tab is not obvious. The Performance Ranking table only lists the middle level of the hierarchy, not individual buildings; at this level of aggregation, irregularities or particularly good/poor performance buildings are washed out, so everything looks good. This may be misleading. More control over what is visible at different levels of a hierarchy of buildings is desirable.
''Baseline / Pulse Adaptive Model'': unlike other EM users interviewed to date, NV does not feel as though the Pulse adaptive model is a black box, and that she understands what interval the baseline is based on.
''On predicting energy savings'': NV would like to know if the energy savings measures implemented at one buildings could also apply to another building, and which measures won't transfer. We discussed the role of simulation and making predictions (such as Goodwin et al's data sculpting idea).
''On the use of Excel'': NV uses Excel to double check what she sees in EM, she uses a template (such as SFU SSB Energy Calculator.XLS, described below) to see the percentage change from year to year over the C.Op period. This could probably be done in Pulse EM. The month-to-month %ੳ most interesting.
''EM Usage Analysis'': NV has no way of knowing who is using the Pulse EM for the projects with which she is involved: who is viewing management charts, reports, creating messages, thresholds, alerts. Should incentives be provided for EM engagement? With enough users actively using the EM management charts and leaving messages, you can effectively crowdsource the interpretation of events in a building. This is similar to what [[MT at SSD is doing|Pulse-MT-13.10.24]] with the Energy Ambassador program.
''Integration'': EM is currently not integrated with building automation software (e.g. Siemens), a problem also identified by [[BE at UBC|Pulse-BE-13.09.05]].
''Customization'': a need for one's own default page, prioritize particular buildings in a portfolio, those of current interest. She remarks that the [[Energent EIMS|http://www.energent.com/technology/dashboard]] is more customizable in this regard but harder to set up.
!!Document Notes
1. Email: Fwd: ~NorthWrite ~ScoreCard for Applied Science Bldg Megajoule on 9_3_2010
*Energy Expert ~ScoreCard for SFU Applied Science building, details 24 hour consumption period, indicating time of peak load, daily OAT high and low, calculated savings, actual vs. expected consumption, a load profile, 5-day weather forecast
2. SFU SSB Energy Calculator.XLS
*colour-coded Excel spreadhseet with tables reporting on a single building at SFU (Square footage at top) with monthly and annual consumption data, peak demand data, average daily consumption indicating baseline periods, C.Op implementation periods, coaching periods, missing data; also reports deltas between 2009-2011; for electricity and hot water; hot water data doesn't inclde peak demand; hot water data includes HDD and consumption intensity (GJ / HDD)
*grouped bar charts comparing last 4 years of monthly electicity consumption, monthly electicity peak demand, hot water consumption
*an unexplained / untitled line chart
*a GJ/HDD line chart comparing past 4 years of hot water consumption intensity
!Why Unremarkable Things Matter
*The story of ''mass observation''
*Overcoming cultural impulses:
#the desire for everything to be the same
#the desire for a good story
#the desire for speed and action
#the desire for closure
>//Pursue truth, not rarity. The atypical can fend for itself...// - essayist Nicholson Baker
Don't ask causal/historical questions too early:
>//People who are constantly asking 'why' are like tourists who stand in front of a building reading Baedeker [an old tourist guidebook] and are so busy reading the history of its construction, etc., that they are prevented from seeing the building.// - Wittgenstein
*The mundane in the remarkable and the remarkable in the mundane
!!Aside: Reflections on my literary tastes
*An appreciation of detail - hysterical realism
*The everyday as unfamiliar, finding the familiar and mundane during unusual / unexpected events
*The absence of the dramatic, the incident, or the necessary elements of genre-fiction is seldom disappointing
!On Finding and Manufacturing Qualitative Data
The author's justification for naturally-occurring or ''found data'': naturally occurring data does not flood the research setting the researcher's own categories. It doesn't put research subjects in the position of the disinterested expert, having to reflect on their own processes and opinions. The topic is directly studied, as opposed to the indirect study of the topic, having to make inferences connecting the manufactured data to the topic of study. It allows for the detection of novel issues, those beyond prior expectations of the topic of study. Finally, the data produced is rich. Nevertheless,
*No data are intrinsically unsatisfactory
*No data are untouched by researchers' hands
*Polarities like ''naturally-occurring data vs. manufactured data'' are rarely helpful if carried too far
*Good quality data does not guarantee good quality research
*Everything depends on how you analyse the data rather than the data's source
>//"The question is not why we should study natural materials, but why should we not?"// - Potter (2002)
!!Manufactured Data in HCI
*The above reflected in reviews of our domestic interruption field study:
>//It seems to me that activities labeled interruption is based solely on the researchers' opinion/interpretation, rather than the informants'// - reviewer
*So how do we proceed? How do we study naturally-occurring instances of interruption and distraction when they are (a) infrequent occurrences, by definition a shifting of attention a from a primary task; and (b) of locally constructed meaning? Could it be possible at least to come to some shared meaning of what an interruption is? what a distraction is?
*All interventionist research in the HCI community: //Polarities like ''naturally-occurring data vs. manufactured data'' are rarely helpful if carried too far// - true especially if we are concerned with studying interventions (critical ethnography / action research), studying a deployed technology in a context for the first time, or during its initial phases of deployment. In a sense, all the observations will be manufactured data, as they wouldn't be naturally occurring had we, as HCI practitioners and designers, not intervened.
!Instances or Sequences?
The author emphasizes the importance of studying sequences situated in shifting contexts, contexts continually reconstructed by those situated in it. The author distinguishes between a focus on how and what and why:
*''Structural ethnography'' - the organization and distribution of subjective meanings (the //whats//)
*''Articulative ethnography'' - how meaning is locally constructed (the //hows//)
*''Practical ethnography'' - interpretations are neither limitless nor purely formal, organizational embeddedness.
Quantitative research, in its focus on causation and correlation, treats the phenomena of causation/correlation as purely operational, part of a chain of events from manipulated inputs to outputs, without describing how the phenomena is locally interpreted. Qualitative research utilizing manufactured data (i.e. open-ended interviews), in an endeavour to tap perceptions of individuals and study responses, they make the phenomena occurring in context unavailable. Qualitative research with naturally-occurring data can study the whats and hows of local interactions, of local meaning of the phenomena. Then, and only then, can we move on to study the whys of how that phenomena is locally embedded.
!!A focus on instances in interviews during the ~C-TOC Interruption study
*We conducted closed interviews and questionnaires, a follow-up to the quantitative data gathering, where participants had to explain their own behaviour during the study.
*How did we proceed? Quantitative analysis, counts of utterances, not embedded in a sequence - the sequence ignored. We only sought to justify or explain our quantitative findings.
!Applying Qualitative Research
Addressing the common criticism of qualitative research (when the research addresses a controversial topic): it's not "real data", or it's just "talking to people", reporting anecdotes, perceptions. Or is it trying to further a hidden agenda? The key strength of ethnographic work, the ability to depict what happens in situ, describing how institutions function, is represented as a weakness. We should focus on ''social settings'', a focus on how institutions are constructed by the observed participants, rather than ''social worlds'', a focus on perceptions as revealed by interviews.
*Fallacy #1: the ''explanatory orthodoxy'': people are puppets of social structures, people's behaviour can be explained by one or more face-sheet variables (social class, gender, ethnicity) - here, the obsession with correlation, cause and effect, allows the phenomenon to escape
*Fallacy #2: the ''divine orthodoxy'': people are dopes, people say one thing in interviews and do another; there is a good practice but no-one follows it. Social scientists can see through people's claims and know better. The belief that there is some normative standard best practice. Questions are asked that never come up in their daily lives.
*//"Researchers who search for a best practice tread a slipper path."// - Who decides what these are? The stakeholders of the research? The researchers?
>//"If we are to understand human-computer interaction, we must go beyond a simple focus on individuals."// - Heath and Luff (//Technology in Action//, 2000) on the situated role of a CAD system and its usage within in an organization. A relentless emphasis on how technologies are used in coordination with colleagues [...] leading to practical inputs far wider than the usual focus with the study of HCI on how one employee uses a machine.
Numbers count with qualitative research, improving research validity: obtaining an sense of variance in the data at an early stage, to guide future research, or at a later stage, having identified a phenomenon, checking its prevalence.
!References
<<cite Silverman2007 bibliography:Bibliography>>
''Quality Metrics'' evaluate original data and/or images generated, and at the interim stages in the [[Information Visualization Pipeline]] (the quality metrics pipeline is a layer over top of this). This evaluation can be automated (algorithms produce quality scores), or by [[Heuristic Evaluation]]. Examples of ''quality metrics'' include the use of image processing algorithms to detect visible clusters, correlation, outliers, complex patterns, in a projection of high-dimensional data, the amount of visual clutter (image quality) in an image, or the degree to which data abstraction disrupts relevant patterns in the data (as a result of sampling, aggregation: feature preservation vs. information loss, original class separation). Metrics serve to create and evaluate alternative mappings or renderings of the data, and help produce optimal final representations.
''Quality Metrics'' can be used at the ''Data Transformation stage'' (feature selection, projection, aggregation, sampling), at the ''Visual Mapping'' stage (considering alternative visual encodings), and at the ''Rendering/View Transformation'' stage (alternative images). Often, complex iteration takes place between these stages, as the user drives and select metrics to create optimal representations, via threshold selection and metrics selection.
!!!!Source:
<<cite Bertini2011 bibliography:Bibliography>>
Inspired by rational choice theory in the field of economics. Selective forces in the environment (task, information environments) optimize the user's behavioural choices in order to maximize their goals. An optimization of the adaptation of behaviour. Requires an understanding of the structure and dynamics of the environment. With this, systems can be designed to carry out this optimization. Also known as Methodological Adaptationism.
Source:
<<cite Pirolli2009 bibliography:Bibliography>> ch. 1 p.22
Source: <<cite Pascoe2007 bibliography:Bibliography>>
!Surprised, yet not
*Micro politics of high school, behaviour of groups of boys is hardly surprising, reminiscent of my own high school experience
*Aside: American high schools are hyper-ritualized? Emphasis on annual events more so than in Canadian schools?
*Fly-on-the-wall methodology - throughout the text (<<cite Pascoe2007>>), I was astonished re: the type of utterances, behaviour Pascoe observed. How much of this was exhibitionist / exaggeration / or reverse Pygmalion effect?
*Appendix helpful: a researcher w/ a neutral stance (gender, age)
*Adversarial ethnography - Pasoce's personal politics and disposition placed teenage boys in an adversarial role. Upon reading the author's appendix it felt as though the experience was one of 'ethnography among the enemy'. A possible bias here? How would the book be different had the author been an older, heterosexual male?
!Ethnography considered harmful
*defining research questions
*researcher tourism (<<cite Crabtree2009>>)
**An emphasis on critical questions in HCI, not merely interpretation. Rather than "how do people use [system x]?", we ask "what are the problems faced with using [system x]?", and "can [system y] outperform [system x]? Is it more useful, usable, easier to learn?" Longitudinal A/B studies?
><<cite Crabtree2009>> acknowledge the increase of new ethnographic approaches reported in the CHI community, particularly for studying contexts outside of the workplace. These approaches have been appropriated by the community based on a misconception of what "work" is. The authors maintain that "work" pertains to actions and interactions with objects in //any// setting, be it the traditional //or// non-traditional workplace, the home, the museum, anywhere. The misconception is that "work" only pertains to workplace activities in workplace contexts, centred around productivity and efficiency, and not to other activities that are shaped by culture and social interaction. These new ethnographic approaches thereby focus too heavily on culture and context, on defamiliarizing activities and interactions with objects into literary abstractions, or on critique of the design process, choosing not to explicitly focus on the "work" being done, the situated actions and interactions. As a result, these methods are appropriate for the social sciences and typical deliverables of this field (being literary in nature), but not for informing design, as ethnographic approaches in the HCI community has historically been used for. Cultural abstractions, de-familiarized practices, design process critique, and exotic stories do not inform design, they are tourism for the HCI researcher.
>The take-home message is that "work" is far more general than what happens in traditional workplaces, and that ethnographic research approaches should above all focus on situated action for the purpose of informing design.
>Appropriate to read for practitioners studying non-workplace contexts, or non-traditional workplaces, workplaces in other cultures; for those studying conventional workplaces, a focus on studying actions and interactions should have never shifted (so you're safe)
>I'd be curious to read about new ethnographic approaches studying traditional workplace settings, and their results.
*Above in response (homage) to <<cite Greenberg2008>>, an earlier call for researchers to reduce our dependence on usability studies as the penultimate evaluation method. A call for the diversification of formative and summative user studies, urging the community to appreciate the rigour involved in qualitative longitudinal research.
*No adversarial ethnography in HCI? Less of an emphasis on "the other", or the foreign user? A focus on marginalized / disadvantaged users, on possible solutions? e.g. studying homeless youth with access to mobile computing.
!Hypothesis generation vs. verification?
*post-positivist perspective: vague research questions addressing high-level constructs (i.e. gender, sexuality, masculinity) vs. specific research questions re: issues tied to these concepts: (i.e. how does compulsive heterosexual behaviour marginalize/alienate certain individuals?) - does the latter still point to an ethnographic study? opportunity to generate hypotheses vs. setting out to verify a hypothesis?
*in HCI, the high-level constructs might be learning, accessibility, cohort differences, while specific research questions tied to these concepts may be: "what are the barriers to usability and accessibility that hinder older adults' learning to use mobile computing?" - does the latter still point to an ethnographic study? or does it begin that way, only to imply a complimentary design and evaluation study?
*Does ethnography with a post-positivist theoretical perspective assume a hypothesis at the outset?
!Response, not intervention
*recent criticism of the CHI community:
**Kosara, R: [[The State of Information Visualization, 2012|http://eagereyes.org/blog/2012/state-information-visualization-2012?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+EagerEyes+%28EagerEyes.org%29]]
>// I don᮴ to end up with a situation like in human-computer interaction, where all the constructive work is done in industry and all academics seem to do is study what others build.//
**Landay, J. A. [[CHI becoming even more irrelevant?|http://dub.washington.edu/blog2010/aug/29/chi-becoming-even-more-irrelevant/]]
*Do we have a mandate to perform action research / interventionist research within HCI? What is the relative contribution of qualitative ethnographic research re: technology, interpretation of culture surrounding technology use?
!References
<<bibliography>>
Tangible information contributing to judgments made in the analytical process, comprising of:
*''Elemental artifacts'': from isolated information sources - source intelligence, relevant information, assumption, evidence
*''Pattern artifacts'': from collections of information - patterns and structure, temporal / spatial patterns
*''Higher-order knowledge constructs'': arguments, causality, models of estimation
*''Complex reasoning constructs'': hypotheses, scenarios
Source: <<cite Thomas2005 bibliography:Bibliography>>
<<list filter [tag[references]]>>
!BELIV' 06:
<<bibliography BELIV-06 showAll>>
!BELIV' 08:
<<bibliography BELIV-08 showAll>>
!BELIV' 10:
<<bibliography BELIV-10 showAll>>
!BELIV' 12:
<<bibliography BELIV-12 showAll>>
!BELIV' 14:
<<bibliography BELIV-14 showAll>>
A curated list of relevant CHI 2012 papers.
!!!!Notes:
*Title links are to the ACM.DL
*WIP posters, case studies, workshop position papers, and alt.chi papers are non-archival (but can be found on the proceedings DVD)
*@@color:#324F17;Green indicates particularly interesting, cool, and/or relevant.@@
*@@color:#666666;Grey indicates that I didn't go the talk / spend time looking at the poster, but I flagged it for later reading.@@
>indicates my commentary from attending the talk
*Here's the [[BibTeX|CHI2012]] for this list
!!!Relating to visualization and sensemaking
<<cite Chuang2012 bibliography:CHI2012 showAll>> - //[[Interpretation and Trust: Designing ~Model-Driven Visualizations for Text Analysis|http://dl.acm.org/citation.cfm?doid=2207676.2207738]]// (Stanford)
>iterative model-driven text visualization; Stanford dissertation browser dataset; radial plots; topical affinity; non-interactive; compares against PCA as benchmark
<<cite Diakopoulos2012 showAll>> - //[[Finding and assessing social media information sources in the context of journalism|http://dl.acm.org/citation.cfm?doid=2207676.2208409]]// (Rutgers)
>presentation FUBAR (no slides); tracking the authority of citizen / local / on-the-ground reporting via social networks
<<cite Dunne2012a showAll>> - //[[GraphTrail: Analyzing Large Multivariate, Heterogeneous Networks while Supporting Exploration History|http://dl.acm.org/citation.cfm?doid=2207676.2208293]]// (MSR)
>lightning-fast talk, slides difficult to read from back of room; complicated interface / interaction technique (deserving of a longer demo/walkthrough); useful for tracing analytical provenance; discrete variables; (Q: how were the tasks determined?) multiple evaluation methods; drag-and-drop visualization creation; co-authorship network and archaeology network datasets
@@color:#324F17;<<cite Eisenstein2012 showAll>> - //~TopicViz: Interactive Topic Exploration in Document Collections// (Georgia Tech / CMU - WIP poster)@@
>visualizing text documents with spring-based layout, interactive support for re-arranging documents; ~LDA-extracted topic tags become nodes in the spring-based layout; supports directed search; intended for academics conducting a literature review; may not scale due to clutter / occlusion (met the author, followed up via email)
<<cite Endert2012 showAll>> - //[[Semantic Interaction for Visual Text Analytics|http://dl.acm.org/citation.cfm?doid=2207676.2207741]]// (VT)
>discussed (and mostly critiqued) in ~InfoVis group meeting: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
<<cite Fisher2012 showAll>> - //[[Distributed Sensemaking: Improving Sensemaking by Leveraging the Efforts of Previous Users|http://dl.acm.org/citation.cfm?doid=2207676.2207711]]// (MSR / CMU)
>//"expert intelligence analysts often use ~PowerPoint"//; conceptual mapping exercises; iterating on previous users' efforts: experiment comparing individual concept mapping vs. 1st iteration collaborative mapping vs. 5th iteration mapping; peer assessment evaluation
<<cite Fisher2012a showAll>> - //[[Trust Me, I'm Partially Right: Incremental Visualization Lets Analysts Explore Large Datasets Faster|http://dl.acm.org/citation.cfm?doid=2207676.2208294]]// (MSR / Southampton)
>can't read slides from back of room; database sampling questions that couldn't be asked previously
<<cite Jianu2012 showAll>> - //[[An Evaluation of How Small User Interface Changes Can Improve Scientists' Analytic Strategies|http://dl.acm.org/citation.cfm?doid=2207676.2208704]]// (Brown)
>controlled study altered UI components and number, impact on quantity and quality of hypotheses generated; hard to read slides (projector / resolution was buggy during this session for several speakers)
@@color:#324F17;<<cite Metoyer2012 showAll>> - //[[Understanding the Verbal Language and Structure of End-User Descriptions of Data Visualizations|http://dl.acm.org/citation.cfm?doid=2207676.2208292]]// (MSR / Oregon - note)@@
>language includes relative layout, spaces as objects; names ﴯvis variables; a describer-interpreter study (collaborative analysis task over a phone line)
<<cite Nagel2012 showAll>> - //Interactive Exploration of Geospatial Network Visualization// (Applied Sciences Potsdam / KU Leuven - case study)
>no zooming, occlusion issues, low-luminance (heavy on aesthetics); conducted open-ended observational user studies at previous conferences
<<cite Muralidharan2012 showAll>> - //A Sensemaking Environment for Literature Study// (UC Berkeley - WIP poster)
>digital humanities / comparing language use across multiple Shakespeare plays - topic discovery / language shift
<<cite Thudt2012 showAll>> - //[[The Bohemian Bookshelf: Supporting Serendipitous Book Discoveries through Information Visualization|http://dl.acm.org/citation.cfm?doid=2207676.2208607]]// (Munich / Calgary - paper + interactivity)
>multiple linked visualizations for exploring a book collection: cover colour, year published vs. period/years written about, author name, keywords, book length / size; very playful, high aesthetic value; in-the-wild observational study, deployed at U Calgary library
@@color:#324F17;<<cite Willett2012 showAll>> - //[[Strategies for Crowdsourcing Social Data Analysis|http://dl.acm.org/citation.cfm?doid=2207676.2207709]]// (Berkeley / Stanford)@@
>user study demonstrates methods for crowd sourcing rigorous analysis: source tracking, directed questions, annotation layers
<<cite Ziemkiewicz2012 showAll>> - //[[Analysis Within and Between Graphs: Observed User Strategies in Immunobiology Visualization|http://dl.acm.org/citation.cfm?doid=2207676.2208291]]// (Brown - note)
>small N, WIP; coded analysis quantitatively using Amar and Stasko taxonomy; between-graphs analysis promotes confidence building
@@color:#666666;<<cite Lee2012 showAll>> - //[[JigsawMap: Connecting the Past to the Future by Mapping Historical Textual Cadasters|http://dl.acm.org/citation.cfm?doid=2207676.2207740]]// (Seoul National / Samsung)
<<cite Gill2012 showAll>> - //~User-Driven Collaborative Intelligence Social Networks as Crowdsourcing Ecosystems// (~ECOdesyn lab - alt.chi)@@
!!!Relating to design
@@color:#324F17;<<cite Heer2012 showAll>> - //[[Color naming models for color selection, image editing and palette design|http://dl.acm.org/citation.cfm?doid=2207676.2208547]]// (Stanford / Tableau)@@
>great talk/slides; DR on the XKCD colour naming dataset; approx. 160 unique colour names; predicting color names from swatches and vice versa; applications in color palette tools
<<cite Kong2012 showAll>> - //[[Delta: A Tool For Representing and Comparing Workflows|http://dl.acm.org/citation.cfm?doid=2207676.2208549]]// (Autodesk)
>extracting and comparing multiple tool tutorials in a single UI
@@color:#666666;<<cite Jansen2012 showAll>> - //[[Tangible remote controllers for wall-size displays|http://dl.acm.org/citation.cfm?doid=2207676.2208691]]// (INRIA)
<<cite Pierce2012 showAll>> - //[[Undesigning Technology: Considering the Negation of Design by Design|http://dl.acm.org/citation.cfm?doid=2207676.2208540]]// (CMU)
<<cite Switzky2012 showAll>> - //Incorporating UCD Into the Software Development Process: A Case Study// (Austin Energy - case study)@@
!!!UX Research methods & tools
<<cite Faste2012 showAll>> - //[[The untapped promise of digital mind maps|http://dl.acm.org/citation.cfm?doid=2207676.2208548]]// (CMU)
>comparing digital mind map tools with good ol' pencil-and-paper methods; possibilities for design
<<cite Kaptein2012 showAll>> - //[[Rethinking statistical analysis methods for CHI|http://dl.acm.org/citation.cfm?doid=2207676.2208557]]// (Eindhoven, ~Heriot-Watt)
>pitfalls of writing results, reviewing: power, confusing p values and effect sizes, strength of Bayesian t-tests / ANOVA
>Also see J. Robertson's May 2012 article in Communications of the ACM: //[[Likert-type scales, statistical methods, and effect sizes|http://dl.acm.org/citation.cfm?doid=2160718.2160721]]//
<<cite Lingel2012 showAll>> - //Ethics and Dilemmas of Online Ethnography// (Rutgers - alt.chi)
>autoethnography - studying your own community (extreme body modification; online recruiting and control of methods through community blogs
@@color:#324F17;<<cite Swearngin2012 showAll>> - //[[Easing the generation of predictive human performance models from legacy systems|http://dl.acm.org/citation.cfm?doid=2207676.2208415]]// (IBM)@@
>free popcorn; java tool that uses unit testing and screen capture technology to automatically generate multiple workflows and task performance models (replaces exhaustive manual method)
<<cite Teo2012 showAll>> - //[[CogTool-Explorer: A Model of ~Goal-Directed User Exploration that Considers Information Layout|http://dl.acm.org/citation.cfm?doid=2207676.2208414]]// (IBM)
>predictive model for novice exploration behaviour
@@color:#666666;<<cite Abeele2012 showAll>> - //Increasing the Reliability and Validity of Quantitative Laddering Data with ~LadderUX// (Leuven / ~LadderUX - WIP poster)
<<cite Antin2012 showAll>> - //[[Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India|http://dl.acm.org/citation.cfm?doid=2207676.2208699]]// (Yahoo / UC Berkeley)
<<cite Correll2012 showAll>> - //[[Comparing Averages in Time Series Data|http://dl.acm.org/citation.cfm?doid=2207676.2208556]]// (UW Madison / Northwestern)
<<cite Dell2012 showAll>> - //[["Yours is Better" Participant Response Bias in HCI|http://dl.acm.org/citation.cfm?doid=2207676.2208589]]// (Washington/ San Jose / MSR)
<<cite Friess2012 showAll>> - //[[Personas and Decision Making in the Design Process: An Ethnographic Case Study|http://dl.acm.org/citation.cfm?doid=2207676.2208572]]// (North Texas)
<<cite Gomez2012 showAll>> - //[[Modeling task performance for a crowd of users from interaction histories|http://dl.acm.org/citation.cfm?doid=2207676.2208572]]// (Brown)
<<cite Jain2012 showAll>> - //Case Study: Longitudinal Comparative Analysis for Analyzing User Behavior// (Google / MSR - case study)
<<cite Ko2012 showAll>> - //Mining Whining in Support Forums with Frictionary// (Washington - alt.chi)
<<cite Kusunoki2012 showAll>> - //Applying Participatory Design Theory to Designing Evaluation Methods// (workshop)@@
!!!UX Research methodology and theory
<<cite Hayes2011 showAll>> - //[[The relationship of action research to human-computer interaction|http://dl.acm.org/citation.cfm?doid=1993060.1993065]]// (UC Irvine - ~ToCHI)
>read in February (see TM email thread re: AR) [[notes|Action Research]]
@@color:#324F17;<<cite Obrist2012a showAll>> - //In Search of Theoretical Foundations for UX Research and Practice// (Newcastle - WIP poster)@@
>collecting and sorting the epistemologies, theoretical frameworks, methodologies of HCI research
@@color:#666666;<<cite Karapanos2012 showAll>> - //Theories, Methods and Case Studies of Longitudinal HCI Research// (workshop)
<<cite Obrist2012 showAll>> - //Theories behind UX Research and How They Are Used in Practice// (workshop)
<<cite Rode2012 showAll>> - //Qualitative Research in HCI// (workshop)@@
!!!Misc
@@color:#324F17;<<cite Davies2012 showAll>> - //[[The Case of the Missed Icon: Change Blindness on Mobile Devices|http://dl.acm.org/citation.cfm?doid=2207676.2208606]]// (UCL)@@
>change and inattentional blindness can occur on a mobile device; 2 user studies - change blindness with disruptions (orientation change, flicker, push notification); inattentional blindness in multiple screen locations (i.e. while game playing)
@@color:#666666;<<cite Tanenbaum2012 showAll>> - //[[Steampunk as design fiction|http://dl.acm.org/citation.cfm?doid=2207676.2208279]]// (SFU SIAT)@@
>@@color:#666666;apparently a talk given in a steampunk costume@@
@@color:#666666;<<cite Tomlinson2012a showAll>> - //[[Collapse Informatics: Augmenting the Sustainability & ICT4D Discourse in HCI|http://dl.acm.org/citation.cfm?doid=2207676.2207770]]// (UC Irvine, Indiana U, Bloomington, Bureau of Economic Interpretation)@@
>@@color:#666666;best paper winner / sustainability prize award winner; from the program: //"Augments the discourse on sustainable HCI and ~ICT4D to include notions of preparation for and adaptation to potential societal collapse, suggesting exemplars for interactivity design in response to such scenarios."//@@
*MUX: CHI / GI / GRAND review session:
**GRAND: Jane ~McGonical TED talk on game design
**GRAND: Chris Anderson (Wired talk on Desktop Manufacturing)
**CHI (Peter): //From competition to metacognition// (Foster et al., UCSD)
**CHI (Matt): //Backtracking Events as Indicators of Usability Problems in ~Creation-Oriented Applications// (Akers et al.)
**CHI (Syavash): //Understanding palm-based imaginary interfaces// (Gustafson, Rabe, and Baudisch)
**CHI (Syavash): //Does activism hurt slactivism?// (Lee and Hsieh, Michigan)
**CHI ~IllumiRoom (Jones, Benko, Wilson (MSR))
**CHI (Matei): //Reasons to question seven segment displays//
**CHI (Mona): ~AnyType (Devendorf, UC Berkeley)
A curated list of relevant CHI 2013 papers.
!!!!Notes:
*Title links are to the ACM.DL
*Here's the [[BibTeX|CHI2013]] for this list
<<bibliography CHI2013 showAll>>
[[Full searchable list of all 493 (!) accepted CHI papers|http://confer.csail.mit.edu/chi2014/papers]]
//Visualizing Dynamic Networks with Matrix Cubes//. Benjamin Bach, Emmanuel Pietriga, ~Jean-Daniel Fekete
//Show Me the Invisible: Visualizing Hidden Content//. Thomas Geymayer, Markus Steinberger, Alexander Lex, Marc Streit, Dieter Schmalstieg
//~Sample-Oriented ~Task-Driven Visualizations: Allowing Users to Make Better, More Confident Decisions//. Nivan Ferreira, Danyel Fisher, Arnd Christian Kong
//Highlighting Interventions and User Differences: Informing Adaptive Information Visualization Support//. Giuseppe Carenini, Cristina Conati, Enamul Hoque, Ben Steichen, Dereck J Toker, James T Enns
//~Task-Driven Evaluation of Aggregation in Time Series Visualization//. Danielle Albers, Michael Correll, Michael Gleicher
//Automatic Generation of Semantic Icon Encodings for Visualizations//. Vidya Setlur, Jock D Mackinlay
//Understand Users㯭prehension and Preferences for Composing Information Visualizations//. Huahai Yang, Yunyao Li, Michelle Zhou
<<bibliography Bibliography-DR showAll>>
<<bibliography Bibliography-GraphicalInference showAll>>
!![Chaney2012] - Visualizing Topic Models
<<cite Chaney2012 bibliography:Bibliography-Overview>> present a system for visualizing topic models, a system which addresses several goals: //"summarize the corpus for the user; reveal the relationships between the content and summaries; and, reveal the relationships across content."//, addressing the need to //"discover and visualize the structure of a collection in order to more easily explore its contents"//.
>//"One of the main applications of topic models is for exploratory data analysis, that is, to help browse, understand, and summarize otherwise unstructured collections".//
They propose a method intended to //"organize, summarize, visualize, and interact with a corpus."//
They conducted a user study in which they asked //"for qualitative feedback on the Wikipedia navigator. The reviews were positive, all noting the value of presenting the high-level structure of a corpus with its low-level content. One reviewer felt it organized similar to how he thinks."//
They released their web-based tool with three demo data sets: [[Wikipedia|http://bit.ly/wiki100]], [[NYT articles|http://bit.ly/nyt-demo]], [[US federal cases|http://bit.ly/case-demo]]; all of which were clean and homogeneous. A sign of adoption: someone had used their open-source code to visualize [[topic models of arXiv|http://bit.ly/arxiv-demo]]
!!!!Comments
*Did not mention tasks in user study
*task words: "explore", "summarize", "organize", "interact" (more //how// than //why//)
!![Cao2010] - ~FacetAtlas
<<cite Cao2010>> present ~FacetAtlas, a visualization tool for presenting document collections as facets, entities, and relations, facilitating analysis at the local and global levels. Facets are groups of entities, and entitites can form relational clusters within facets, they can also form relations with entities in other facets, reflecting the "multi-faceted nature of document collections and the correlations that exist between documents".
>//"owever, be- cause of information lost when projecting from a high dimensional space to 2D coordinates, it is often hard for users to understand the semantic meaning of the resulting clusters.[ץ combat the information lost due to dimensionality reduction by providing a novel multifaceted graph-based display that is integrated with an optimized density map."//
They evaluate ~FacetAtlas using case studies and a controlled user study:
>//"Note that both cases were suggested by the medical experts who used our system."//
Lots of data formatting / wrangling required before it is compatible with ~FacetAtlas.
Recruited na堰articipants in a straw-man comparison study (as in <<cite Liu2013>>'s evaluation of Newdle). Conjectured some tasks that were "based on our experts' recommendations":
*T1: identify the major clusters for a test query;
*T2: identify the representative members in the clusters for the query;
*T3: identify the facet with the most within-cluster connections;
*T4: identify the facet with the most cross-cluster connections;
*T5: identify the facet of the two most connected members on the symptom facet;
*T6: identify the facet with the most overall connections across entities.
Metrics tested were time and error, as well as subjective opinions of usability and utility of the various features and interactions.
!!!!Comments
*㮱: what is "life: each entity and relations are assigned"
*Conjectured tasks in user study
*"Explor[e/ation/ing]" the task, also "unstructured search", "interactive exploration"
*screen capture video of case study no longer available on author's academic web page
*homogeneous, clean document collections (~PubMed documents)
*usage scenario not case study: "case study" suggested by domain experts (in interview? how long? how much feedback collected?), but performed by authors
*expert interviews appear last in the paper, but when did they occur chronologically? before/after user study? before/after case studies? Are these users (physicians) actually representative of the system's intended users?
!![Hevner2004] - design science in information systems research
IS systems research is informed by both behavioural science and design science research: the former cares about truth, the latter about utility, a feedback loop. It is difficult to study the application of IT to areas that "were not previously believed to be amenable to IT support". IS behavioural research focuses on artifact's use, intention to use, perceived usefulness, impact / net benefits, and information quality. IS design science research focuses on quantitative (comparisons with other designs, optimization proofs, simulations) and empirical/qualitative evaluations.
Goal of this paper is to provide guidance on conducting, evaluating, and presenting design science research. Design described as process and as artifact, both verb and noun, as a process: a build and evaluate loop, often repeated several times, an evolution of process and artifact; as an artifact: constructs (a language for problem and solution definition), models (representations of the world using constructs), methods (processes for navigating the solution space), and instantiations. The generate / test cycle: generate alternative designs and test alternatives against requirements/constraints.
Environments include people, organizations, technology, having a business need that motivates IS research. IS research involves a develop/build and justify/evaluate loop: assess (from build to evaluate) and refine (from evaluate to build); the result is application to the environment and contributions to IS research in the form of foundations (theories, frameworks, instruments, constructs, models, methods, instantiations) and methodologies (data analysis techniques, formalisms, measures, validation criteria); these foundations inform IS further research.
Routine design is different from design science research: the former uses known foundations, while the latter approaches "wicked problems" in unique or innovative ways or approaches solved problems in more efficient ways. In early stages of a discipline, all artifacts are experiments until design-science research results are codified and best practices formalized.
Wicked problems involve unstable requirements, ill-defined contexts, complex interactions between solution and problem, malleable processes and artifacts, dependence upon creativity, or a dependence on teamwork. Design is in a constant scientific revolution.
Authors propose guidelines for design science research:
*1. design as an artifact: must produce construct, model, method, instantiation
*2. problem relevance: must present solution to problem
*3. design evaluation: evaluate utility, quality, efficacy of solution
*4. research contributions: clear and verifiable contributions in areas of design artifacts, design foundations, design methodologies
*5. research rigor: in both construction and evaluation (though overemphasis on rigor can reduce relevance)
*6. design as a search process: using available means to reach solution, satisfice if necessary
*7. communication of research: presented effectively to technology- and management-oriented audiences.
Also categories evaluation types for evaluating functionality, completeness, consistency, accuracy, performance, reliability, usability, fit w/ organization, assessment of style, etc, matched appropriately to artifact and metrics of interest:
*1. observational: case study / field study
*2. analytical: static analysis, architecture analysis, optimization, dynamic analysis
*3. experimental: controlled experiment, simulation
*4. testing: functional (black box), structural (white box)
*5. descriptive: informed argument, scenarios (only for innovative artifacts for which other forms of evaluation are infeasible)
Presents 3 case studies w.r.t to proposed guidelines.
It's nice to know why a solution works, but it's more critical to know that //it does// work and how well it works, and to characterize the situations in which is does work. Iteratively identify deficiencies and address them before analyzing why the rest of the artifact work. How to define optimal solution within a search space is difficult.
Recognizes a lag between academic research and adoption, and also the apparent ad hoc nature of solutions developed in industry. Encourage more industry/academic collaboration.
Research challenges in design science research include building a theoretical base, a need for constructs, models, methods, tools, problem documentation, keeping pace with technological development, and a need for rigorous evaluation methods.
!!!!Comments
*recommended reading from B. Shneiderman to DSM authors
*also see 2013 follow-up paper
*MS's notes:
>"I found the notion of 'IT artifacts [being] defined as
>- constructs (vocabulary and symbols),
>- models (abstractions and representations),
>- methods (algorithms and practices), and
>- instantiations (implemented and prototype systems).'
>...very interesting and helpful. It even reminded me a bit of Tamara's Nested Model. Now I also want to go back and read the original March and Smith 1995 paper, and of course Hevner's 2013 paper. I'm very curious! My reading list for after Mar 31 starts to pile up :)"
!References
<<bibliography>>
<<bibliography Bibliography-Pulse showAll>>
<<bibliography Bibliography showAll>>
!CPSC 533C: Special Topics in Graphics - Information Visualization (Fall 2009)
<<bibliography Bibliography-CPSC533C showAll>>
!PSYC 579: Special Topics in Perception - Visual Display Design (Winter 2011)
<<bibliography Bibliography-PSYC579 showAll>>
!EPSE 595: Introduction to Qualitative Research (Winter 2012)
<<bibliography Bibliography-EPSE595 showAll>>
!Evaluation of Information Visualization & Visual Analytics Systems
<<bibliography Bibliography-ToRead-VisEval showAll>>
!On Qualitative Research: Methods and Methodologies
<<bibliography Bibliography-ToRead-Eval showAll>>
!Evaluation Methods in CSCW
<<bibliography Bibliography-ToRead-CSCWEval showAll>>
!Visual Analytics, Sensemaking, & Information Retrieval
<<bibliography Bibliography-ToRead-VASensemaking showAll>>
!Creativity & Cognition
<<bibliography Bibliography-ToRead-C&C showAll>>
!On Design
<<bibliography Bibliography-ToRead-Design showAll>>
!Misc. HCI
<<bibliography Bibliography-ToRead-HCI showAll>>
!Misc.
*[[Tableau Software's Pat Hanrahan on "What Is a Data Scientist?"|http://www.forbes.com/sites/danwoods/2011/11/30/tableau-softwares-pat-hanrahan-on-what-is-a-data-scientist/]]
*[[8 Insights About The Coming Era Of Interactive Design|http://www.fastcodesign.com/1671611/8-insights-about-the-coming-era-of-interactive-design]] - via ~FastCo
*[[Reuters' Connected China|http://connectedchina.reuters.com/]]
*[[Procedures used by NSA to minimize data collection from US persons: Exhibit B 쬠document||http://www.guardian.co.uk/world/interactive/2013/jun/20/exhibit-b-nsa-procedures-document]]
*[[FAQ: What You Need to Know About the NSA Surveillance Programs|http://www.propublica.org/article/nsa-data-collection-faq]] by J. Stray (~ProPublica)
*[[Data visualization: A view of every Points of View column|http://blogs.nature.com/methagora/2013/07/data-visualization-points-of-view.html?WT.mc_id=TWT_NatureMethods]] - Nature Methods column
*[[How A 'Deviant' Philosopher Built Palantir, A CIA-Funded Data-Mining Juggernaut|http://www.forbes.com/sites/andygreenberg/2013/08/14/agent-of-intelligence-how-a-deviant-philosopher-built-palantir-a-cia-funded-data-mining-juggernaut/?utm_content=buffercb18d&utm_source=buffer&utm_medium=twitter&utm_campaign=Buffer]] - Forbes
*[[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] by //You Are Not So Smart//
*[[Manuel Lima|http://www.visualcomplexity.com/]]'s [[Information Visualization Manifesto|http://www.visualcomplexity.com/vc/blog/?p=644]]
*[[Geer on cybersecurity|http://geer.tinho.net/geer.blackhat.6viii14.txt]]
!References
<<bibliography VIS2014 showAll>>
!~InfoVis
!!!recommended:
!!!!read:
@@color:#444bbb;''techniques'': <<cite Steinberger2011 bibliography:VisWeek2011 showAll>> - //~Context-Preserving Visual Links// - ''best paper winner'': [[notes|Information Visualization: Techniques]]
''evaluation'': <<cite Borkin2011 bibliography:VisWeek2011 showAll >> - //Evaluation of Artery Visualizations//: [[notes|Information Visualization Evaluation: Quantitative Methods]]
''evaluation'': <<cite Lloyd2011 showAll>> - //long-term case study of ~GeoVis users//: [[notes|Information Visualization Evaluation: Qualitative Methods]]
''theory and foundations'': <<cite Hullman2011a showAll>> - //Benefiting ~InfoVis with Visual Difficulties// - ''honourable mention'': [[commentary|https://docs.google.com/viewer?url=http%3A%2F%2Fwww.perceptualedge.com%2Farticles%2Fvisual_business_intelligence%2Fvisual_difficulties.pdf]]: [[notes|Information Visualization Evaluation: Meta-Analysis]]
''theory and foundations'': <<cite Bertini2011 showAll>> - //Quality Metrics in HDD vis//: [[notes|Information Visualization Evaluation: Meta-Analysis]]
''evaluation'': <<cite Rodgers2011 showAll>> - //Ambient & Artistic Visualization//
''theory and foundations'': <<cite Wickham2011 showAll>> - //Product Plots//
''systems and frameworks'': <<cite Bostock2011 showAll>> - //D^^3^^ toolkit//@@
!!!!to read:
''applications'': <<cite Wood2011 showAll>> - //~BallotMaps// - [[comments|http://www.visualisingdata.com/index.php/2011/10/visweek-updates-by-jerome-cukier-day-5/]]:
>//"[the] authors [...] wanted to see whether the order in which names appear in a voting ballot have an influence on the voter choices. There findings confirmed this, and also that candidates with non-English/Celtic names garnered less votes than expected."//
!!!of interest:
''evaluation'': <<cite Isenberg2011a showAll>> - //Dual Scale Data Charts//
''evaluation'': <<cite Walny2011 showAll>> - //Visualizations On Whiteboards//
''techniques'': <<cite Bae2011 showAll>> - //Quilts for the Depiction of Large Layered Graphs// - [[comments|http://infosthetics.com/archives/2011/11/most_interesting_papers_at_infovis_visweek_2011.html]]:
>//"[..] presented a new way to visualize trees [...]. Rather than the typical node-leaf approach, Bae uses a row-column way of looking at the structure. Personally, I found the learning curve to be very high, but then pretty easy to use. In some ways it seems to introduce visual clutter, but it also feels easier to navigate."//
''techniques'': <<cite Brandes2011 showAll>> - //Asymmetric Relations in Longitudinal Social Networks// - [[comments|http://infosthetics.com/archives/2011/11/most_interesting_papers_at_infovis_visweek_2011.html]]:
>"// used balance to visualize dyadic relationships. Even in its most basic form, a 'Gestaltline' shows type, extent, and time of the relationship. Color is left as a degree of freedom to encode other variables. Using a sparkline or multivariate glyph approach, a gestaltline can easily be placed within text as a dataword. The technique seems like a very intuitive way of viewing relationships."//
''applications'': <<cite Albers2011 showAll>> - //Sequence Surveyor// - [[comments|http://www.visualisingdata.com/index.php/2011/10/visweek-updates-by-jerome-cukier-day-5/]]:
>//"[...] a very impressive tool to explore and compare genomes. [...] a balance of design and function. And for those of us who are not obsessed with genetics, the last slides suggested the tool could be used for literature analysis or to study datasets like the google N-grams."//
''systems and frameworks'': <<cite Albuquerque2011a showAll>> - //Synthetic Generation of High-Dimensional Datasets// - [[comments|http://www.visualisingdata.com/index.php/2011/10/visweek-updates-by-jerome-cukier-day-5/]]:
>//"Simply put, what this does is let a user ataset with a few mouse strokes. Let尰ose you want a dataset with data following a certain pattern, or with certain correlations: with this interface you can easily generate data that approach the form you want. This is plotting data崠in reverse."//
''multidimensional visualization'': <<cite Cao2011 showAll>> - //Multidimensional Clusters//
''multidimensional visualization'': <<cite Turkay2011 showAll>> - //Brushing Dimensions//
!VAST
!!!recommended:
''foundations of the analysis process'': <<cite Kang2011 showAll>> - //Intelligence Analysis Process, Longitudinal Study//
''tree, network analysis'': <<cite Heer2011 showAll>> - //Orion//
''applications'': <<cite Wang2011 showAll>> - //Framework for Designing Visual Analytics System in Organizational Environments// (like MS's paper, more complicated framework)
''posters'': <<cite Anand2011 showAll>> - //Random Projections// - ''best VAST poster''
!!!of interest:
''foundations of the analysis process'': <<cite Albuquerque2011 showAll>> - //Visual Quality Measures//
''foundations of the analysis process'': <<cite Kwon2011 showAll>> - //Roadblocks for Novice VA Users//
''sensemaking and collaboration'': <<cite Ziemkiewicz2011 showAll>> - //Locus of Control//
!~SciVis
''posters'': <<cite Thorson2011 showAll>> - //Designerᰰroach to Scientific Visualization//
!~BioVis
<<cite Patro2011 showAll>> - //~MDMap//
<<cite Smith2011 showAll>> - //~RuleBender//
!LDAV
<<cite Fisher2011 showAll>> (MSR) - //Incremental, Approximate Database Queries and Uncertainty for Exploratory Visualization//
!References
<<bibliography VisWeek2011 showAll>>
<<bibliography Bibliography-Vismon showAll>>
!!Literature Notes
<<cite Sandelowski1998 bibliography:Bibliography-EPSE595>>: ''writing a good read'':
*choose your format with care, no one narrative style fits all
*thick description and descriptive / analytic excess a fine line
>by trying to retell everything, writers end up showing nothing
*''grounded theory'': presentation to emphasize theoretical reformulation of data, with data acting as support
*authorial presence and power: choosing between objective and omniscient views; former more appropriate for results, latter more appropriate for discussion; at least don't mix within sections
*who is speaking? the participant or the researcher?
*on ''metaphors'': a way of knowing - take them seriously; don't mix them (sends mixed messages), do not abort or follow through with details of the metaphor (aborts the process of discovery), or use metaphors that don't fit
*on re-presenting data: maintain order - a ''temporal order'' (researcher or participant perspective, emphasize causality, continuity/discontinuity), a ''prevalence order'' (central tendencies, ranges, exceptions.
*coding families, sensitizing concepts for developing / testing theories
**''six C's'' around a core variable: causes, contexts, contingencies, consequences, covariances, conditions
**''typology family'': comparisons between, within types
**''strategies'': denoting actions, maintains a consistent grammar and word structure
*Additional references:
**Becker 1986, Chenail 1995, Coffer 1996, Eisner 1981, Lofland 1995, Richardson 1994, Tierney 1995, van Maanen 1988, Wolcott 1990, Wolcott 1994
<<cite KadriZald2004>> - on the ''photo essay''
<<cite Baff1997>> - //Realism and Dead Dudes//: representing data as poetry
!!Commentary
*is a ''taxonomy'' a coding family? how does it differ from a ''typology''? Is this a legitimate coding family to use?
*in the [[|DRITWHDD-DR Ethnographic Project]] project, were we consistent about whose voice we were using, in our section on usage examples, were we objective, omniscient, or an unholy mix of both?
*re: photo essays - have these been used in the CHI community? sustainability and HCI come to mind, as does adaptable / adaptive UI, accessibility issues
*re poetry: consonance between form and substance, writing poetry (or using other art forms) as a reflexive practice;
**if the substance in my case is the visual display of data in visualization, use of the info graphic or visualization as the form of presentation is justified? In a way, we did this in the [[|DRITWHDD-DR Ethnographic Project]] with our figures, illustrating the taxonomy we constructed; drawing and diagramming are our reflexive practices: flow charts and taxonomies, decision trees;
**the possibility of using MDS and visualization (as form) in qualitative data analysis where the substance is about the use of MDS (and other forms of DR) for visualization.
!!References
<<bibliography>>
!!Current Research
Our current research on high-dimensional data analysts and researchers is often collaborative, confidentiality and privacy are not always necessary, as it is often the case that those we study are co-authors or identified in our acknowledgements. In large company settings, it is more likely that participants identities are anonymized, but there are likely to be no/little concern for personal privacy of employees, but rather a institutional or enterprise level confidentiality, a protection of intellectual property of the company.
!!First Nations
Tangential to my involvement in the ~C-TOC project during my Master's, there was a project related to the cross-cultural acceptability of the tool: several focus groups of health workers representing several ethnic communities were held. I recall there being difficulty involving First Nations communities. At the time I did not comprehend why there was any difference between this group and other ethnic demographics. I interpreted this as animosity towards research in general. Upon reading the ~Tri-Council chapter on research with First Nations, I'm now aware that the expectations towards research and the dissemination of research findings is different among First Nations people, thus explaining the additional issues and barriers the ~C-TOC group faced when considering involving First Nations communities.
!!Children
Re: studies involving children - MR, CT, and JM, (and also Inkpen (1996)) seem to have found a unique workaround to studying children: observing and interviewing children on a short-term one-time basis while they accompanied parents to a public place, a museum (Science World). I'm not sure to what extent were children viewed from a social constructivist perspective in this study. Parental involvement and consent was necessary. Children were free to leave/discontinue the observation session at any time.
''Sources'': <<cite Tri-Council2010 bibliography:Bibliography-EPSE595>>: ch. 9-10, <<cite Freeman2009>>: ch. 2, 3, 5
In November I joined a research project here in the computer science department, where the research team is currently trying to explain and defend their qualitative research methodology (post-hoc).
It is often the case that in my field, human-computer interaction, most qualitative research methodologies get lumped under the banner of grounded theory. However now I am aware, thanks to our discussions and readings in the course, that this is not true in our current case. This project began with a well-defined causal hypothesis, which at the abstract level could be stated as: "IF someone is faced with scenario X, THEN they perform process Y". Furthermore, the goal of the project was to find an objective truth that generalized to a range of contexts and domains. Data collection and analysis were concurrent, the former by means of interviewing and collecting found data, and the latter by means of iterative summarization and keyword extraction, seeking evidence for and against the hypothesis.
In short, parts of the methodology were very much [[Grounded Theory|Grounded Evaluation]]-flavoured, however the over-arching epistemology was objective in nature, and data analysis was both top-down (hypothesis-driven) and bottom-up (data-driven). There's been some debate in our group as to whether our methodology was more closely aligned with [[Analytical Induction]] (Znaniecki, 1934), which to our understanding is an positivist qualitative methodology that predates grounded theory. But it's not a perfect fit either - we didn't actively seek out deviant cases in our data collection. It also appears to pre-date post-positivist thought, and as such doesn't handle uncertainty well.
Is [[Analytical Induction]] still considered a legitimate methodology? Or does our methodology fit more closely with a top-down variant of [[Grounded Theory|Grounded Evaluation]]?
>I don't think you can have a top down grounded theoryᴠjust violates the fundamental idea of what grounded theory is.
[...]
>Analytic induction is a generic term that isn't much used because we now have much more clearly defined methodologies౹34 the idea that one might work from the data to theory was novel and indeed if my memory serves me (which it doesn't always) the idea was really borrowed from field studies like biology, i.e., using observations to generate hypotheses. So, the idea of AI is fundamental to all interpretive methodologies and most critical methodologies.
[...]
>I'm not a huge fan of mixed methods studies because most typically they are multiple methods studies (using a variety of methods within either a post-positivist or social constructivist epistemological framework), but the study you describe might actually be an example of a more fundamental 'mixing.' In mixed methods one or the other can be the dominant framework or they can be on an equal footing. There is also a temporal dimension-does one or the other paradigm precede the other or are they some how simultaneously manifest in the research. There are lots of good resources on mixed methods including the Journal of Mixed Methods Research and a very good reference on mixed methods is the SAGE Handbook of Mixed Methods in Social & Behavioral Research by Abbas Tashakkori & Charles B. Teddlie.
<<list filter [tag[projects]]>>
!![Zimmerman2010 ]
Research through Design (~RtD) is analyzed and critiqued in <<cite Zimmerman2010 bibliography:Bibliography>>'s DIS '10 article. ~RtD borrows from design practices and particularly employed when approaching difficult "//wicked problems//" (i.e. problems with societal change, [[Designing for Appropriation]]. There is debate in the HCI community as to whether ~RtD constitutes a viable theory and legitimate scientific method of inquiry, as it tends to lack documentation, theoretical scaffolding, and the theory generated is often not an explicit goal of the project at the outset. It draws from ethnographic and [[Action Research]], thereby being qualitative research.
>//"~RtD is about research on the future"//
HCI encompasses psychological theory, engineering theory, anthropological theory, theory regarding the holistic generation of artifacts. The authors ask if design theory has a place in HCI research. They interview leading HCI practitioners and summarize past ~RtD projects.
They also summarize the critique of ~RtD: it is not scientific in that it is irrational, not logical or transparent or rigorous, and is poorly documented. ~RtD does usually lead to theory development despite not being an intent of the research group.
The authors call for the adoption of a more rigorous methodology and documentation protocol for researchers, and a consensus regarding how ~RtD projects and theory can be evaluated and critiqued. Furthermore, a venue for discussing ~RtD theory in the form of essays and opinion articles is needed.
!!!!Comments & Questions
*Discussed at MUX forum Nov. 2, 2011
*what are //designerly activities//?
*the boundaries between "research about design" (understanding design), "research for design" (improving design practices), and "research through design" (iteratively designing artifacts for future use) aren't entirely clear. Can't the outcome of a single project theoretically address all three?
*~RtD is a formative approach? Applied in dearly design stages?
*Can you conduct quantitative interviews?
*Does qualitative research (nascent theory) always develop into quantitative? What has happened historically to ethnographic and [[Action Research]]?
*~RtD as a legitimate practice gets more funding and acclaim in Europe. Why?
*~RtD seems like an exploratory process, exploring a previously un-navigated design space, or navigating a design space where there is no (explicit) user requirements (is this [[Designing for Appropriation]])?
!References
<<bibliography>>
Seattle Westin, Thursday August 7, 2014
>//Lectures by [[Jonathan Corum, Maria Popova, Randall Munroe, Edward Tufte|http://style.org/stdp2/]]: world-class design architectures for information, data, images, videos, diagrams, interfaces, presentations.//
[[raw sketchy notes|https://dl.dropboxusercontent.com/u/6397998/STDP-08.07.14/STDP-08.07.14.pdf]] (4.1 MB PDF)
"Real artists ship" - ET quoting Steve Jobs
!!Jonathan Corum (JC / NYT)
>//Jonathan Corum is the science graphics editor at The New York Times and founder of the design studio [[13pt|https://twitter.com/13pt]]. In doing more than 1,000 graphics for the Times, his work emphasizes the clear explanation of complex information.//
*See
**Exploiting/harnessing the human capacities for pattern recognition and learning new patterns: "train until your eyes can see it"
***Example: [[XKCD's Kerning|http://xkcd.com/1015/]]
***learning to trust your eyes; example: discerning between queens, workers, and drones, helpful diagrams in [[Bumble Bees of North America: An Identification Guide|http://www.amazon.com/Bumble-Bees-North-America-Identification/dp/0691152225]]
**Seeing, learning, and appropriating what others have done:
***mbostock's [[Epicyclic Gearing|http://bl.ocks.org/mbostock/1353700]] inspired [[NYT's A Century of Cicadas|http://www.nytimes.com/interactive/science/a-century-of-cicadas.html?_r=0]]
**Look at //more// than you can [see / use]
*Think
**Find a clear thought and get in on paper / on the screen
***Sketching as visual problem solving; sketches are not commitments, they serve to drive toward the answer to the question 'what are you trying to do?'; ugly is fine; [[NYT's Districts Across the Country Shift to the Right|http://query.nytimes.com/gst/fullpage.html?res=9903E6D81E39F937A35752C1A9669D8B63]] began as a crude sketch ([[image|http://3.bp.blogspot.com/-5i3uTwBnkgk/TZI-kZ5saWI/AAAAAAAABl0/5FvX02kBLOo/s640/NYTimes-midterm-results-2010.jpg]]), same with [[NYT's Cassini mission graphic|http://www.nytimes.com/imagepages/2010/04/20/science/space/20cassini_graphic.html?ref=space]]
**Now we sketch in the browser: [[NYT/mbostock's 512 Paths to the White House|http://www.nytimes.com/interactive/2012/11/02/us/politics/paths-to-the-white-house.html]] took 265 iterations; a screenshot of images from git commits by mbostock this week
**Find something your brain recognizes, remember the //aha!// moment, communicate that understanding
*Design
**//good design is clear thinking made visible// - ET
**A case study and redesign of [[a cryptic NYC Amber Alert text message|http://www.nydailynews.com/news/national/amber-alert-wakes-thousand-new-york-sparking-debate-article-1.1401466]] that arrived at 4am: what if your design woke up a city?
**Don't be your own audience. remember what it's like to understand
***A case study and redesign of figures from an academic paper into the [[NYT infographic on the lunge feeding of humpback whales|http://www.nytimes.com/imagepages/2007/12/10/science/20071211_WHALE_GRAPHIC.html]]
*Don't collect trivia: recalls visiting the [[Pop Chart Lab|http://popchartlab.com/]] pop-up shop that sold posters of infographics; avoid the infographic language of nouns, examples, and lines (e.g. types of dogs, coffee, shoes, etc.)
***If news articles were written like infographics, they would read: "[[here are some words. hope you find something|http://www.slideshare.net/openjournalism/amanda-cox-visualizing-data-at-the-new-york-times]]" (images by Amanda Cox)
***Find and show meaningful patterns: [[NYT's Guantᮡmo detainees|http://13pt.com/projects/nyt110425/]]
***Avoid the language of infographics, they sexy/futuristic (example: a [[dashboard in the 2013 film Oblivion|http://static2.businessinsider.com/image/51e43e52ecad04b441000002-1200/vikas-main-display-features-a-prominent-map-for-guiding-jack-a-section-for-monitoring-the-fuel-and-repair-status-of-their-various-drones-a-hydro-rig-monitor-and-finally-a-section-for-maintaining-communications-with-their-headquarters-the-tet.jpg]]), the beautiful yet meaningless or ineffective (example: [[Germany's Big Six-O|http://www.ingraphics.info/germanys-big-six-o/]]); visualization ࠣommunication
**Be skeptical of vanity data; example: a year's worth of day-care report cards on JC's son's diaper changes
**Facilitate fair comparisons: [[NYT's Map of Baseball Nation|http://www.nytimes.com/interactive/2014/04/24/upshot/facebook-baseball-map.html]] vs. # Facebook likes for the Yankees or Red Sox without respect to fan geography
**Design iteration and sketching led to [[NYT's Curiosity rover progress|http://13pt.com/projects/nyt130806/]] + use of composite imagery, maps, timelines, text annotation
*Produce
**Working under restrictions elicits creativity: video broadcast restriction for the Olympics forced JC + NYT to work without video, to [[use composite imagery instead|http://www.nytimes.com/interactive/2014/02/19/sports/olympics/olympics-frame-by-frame.html]]
**Annotation layer more important than data layer. examples: redesign of academic paper on [[whether fleas jump with their feet or with their knees|http://www.nytimes.com/imagepages/2011/02/11/health/20110215_flea.html?ref=science]]; [[NYT's Fairfield on Driving Safety, in Fits and Starts|http://www.nytimes.com/interactive/2012/09/17/science/driving-safety-in-fits-and-starts.html]]
**Learning and sketching in D3.js with the [[NASA Kepler exoplanet data|http://exoplanetarchive.ipac.caltech.edu/]] and the [[eventual interactive graphic done in D3.js|http://www.nytimes.com/interactive/science/space/keplers-tally-of-planets.html?smid=tw-share]]
**NYT Graphics Editor M. Ericson "we are not designers"; JC: "actually, I am"
**Ruthlessly apply common sense
**JC's toolkit: [[BBEdit|http://www.barebones.com/products/bbedit/]] + [[iA Writer|http://www.iawriter.com/mac/]] for mobile design; D3.js + coffeescript for development; Python + [[IPython Notebook|http://ipython.org/notebook.html]] + [[PANDAS|http://pandas.pydata.org/]] for data analysis, [[Alfred|http://www.alfredapp.com/]] for file search, Illustrator and Photoshop
*See + think (understand), design + produce (explain)
**Strive for accuracy, clarity, legibility, empathy, and simplicity (but not simplistic); understanding, elegance (and possibly beauty) may emerge, though beauty isn't that important, but nice to have
***Theoretical Physicist Andrei Linde on the experimental verification of his inflation theory following the big bang: "what if I am tricked? what if i believe in this just because it is beautiful?"
*Design for someone else, respect the reader/viewer
**Example of new collaboration with NYT video department re: [[hexagonal hurricane at Saturn's north pole the size of 4 Earths|http://13pt.com/projects/outthere3/]]
*[[@dragonc's notes|https://www.evernote.com/shard/s6/sh/98b64b1f-f21b-4b97-8956-4e10f334dfec/6dfe456b179234b287ab4e2b61367489]]
!!Maria Popova (MP / brainpickings.org)
>//Maria Popova is the founder and editor of Brain Pickings ([[brainpickings.org|http://www.brainpickings.org/]]), an inventory of cross-disciplinary interestingness spanning art, science, design, history, philosophy, psychology, and more. She has written for Wired UK, The Atlantic, Nieman Journalism Lab, The New York Times, Smithsonian Magazine, and Design Observer, among others, and is an MIT Fellow. She is on Twitter as [[@brainpicker|https://twitter.com/brainpicker]].//
*Concerned with the gaps between information and knowledge, knowledge and wisdom
**Higher education and the rote memorization of information, the failure of eliciting curiosity, turning information into wisdom
*Concerned with //combinatorial creativity//; Lego as combinatorial creativity ([["Yellow" by Nathan Sawaya|http://brickartist.com/gallery/yellow/?tag=lego-art]])
*A case for the user base of one: [[instapaper|https://www.instapaper.com/]]
*Reflections on 7 years of [[brainpickings.org|http://www.brainpickings.org/]] (also at [[holstee.com/brainpickings|http://www.holstee.com/products/brain-pickings-poster]])
#Allow yourself the uncomfortable luxury of changing your mind
#Do nothing out of guilt, for prestige or status, for money or approval alone
#Be generous with your time and your resources and with giving credit, and especially with your words
#Build pockets of stillness into your life: //in NYC, one lives in a state of hyper-vigilance// - ET
#when people try to tell you who you are, don't believe them
#Presence is far more intricate and rewarding an art than productivity (avoid the cult of productivity)
#Expect anything worthwhile to take a long time
*The presence and persistence of just "showing up"; Mary Oliver on rhythm as //body-heaven//, similar quotes on the fallacy of inspiration from Tchaikovsky, EB White, Close: "//inspiration is for amateurs//", Allende: "//show up, show up, show up, and eventually, the muse shows up too//"
*Recent collaboration with [[Georgia Lupi|https://twitter.com/giorgialupi]] of [[accurat|http://www.accurat.it/]] and [[Wendy Macnaughton|http://wendymacnaughton.com/]] to [[examine sleep science research, sleep habits and the creative output of artists/writers|http://www.brainpickings.org/index.php/2013/12/16/writers-wakeup-times-literary-productivity-visualization/]]
!!!Q+A
*[[EB White's Here is New York|http://www.amazon.ca/Here-New-York-E-B-White/dp/1892145022]]
*MP's daily rhythm: wake up early, meditate 25 min, respond to urgent email, read at gym, write (uninterrupted, 2-3 posts / day), email/FB/twitter, deliberate unproductive time (yoga, attend a performance), sleep
*Millenials' FOMO? A: building pockets of stillness, avoiding cult of productivity and the need to stay connected
*[[Carl Sagan's Bologna Detection Kit|http://www.carlsagan.com/index_ideascontent.htm]]
*Q: Any lessons on work/creativity from unsavory people? A: Duke Ellington
**ET in defense of Duke Ellington: [[Duke Ellington & Ray Brown: Fragmented Suite for Piano and Bass|https://www.youtube.com/watch?v=CVP2Zd5E7p8]]
**MP's counterargument: films of Woody Allen tainted by allegations of abuse (ET disgrees, that art stands by itself, apart from the artist)
**ET on [[John Gray's Straw Dogs|http://www.amazon.ca/Straw-Dogs-John-Gray/dp/1862075964]], [[The Silence of Animals|http://www.amazon.ca/The-Silence-Animals-Progress-Modern/dp/0374229171/ref=pd_bxgy_b_text_y]]
*[[@dragonc's notes|https://www.evernote.com/shard/s6/sh/91a1fd9b-a5ee-4eec-a354-d685387ef363/af37dffffcd1eddfb66d954e6321d5c3]]
!!Randall Munroe (RM / XKCD)
>//Randall Munroe is a webcomic author, former NASA roboticist, and the creator of the webcomic [[xkcd|http://xkcd.com/]]. He and the webcomic have developed a cult following. In 2013, a main-belt asteroid (4942 Munroe) was named after him.//
*XKCD "went viral" before "went viral" was a term; surge in popularity followed [[XKCD's Sandwich|http://xkcd.com/149/]]
*Why RM left NASA: contract ended / also: an incident involving a rolling chair, a slick floor, an autonomous robot, an ethernet cable as lasso, and the robot's "wander mode".
*[[XKCD's Movie Narrative Charts|http://xkcd.com/657/]] were envisioned 10 years before XKCD; RM had dreamed of computers extracting this information from scripts / novels (he later realized this was computationally very challenging, so he just drew them instead)
*"I'm a chronic explainer"
*"Put in some jokes". Jokes make whatever information you are showing somewhat disarming (though jokes are always afterthoughts of his infographic work)
*Beware the trap of efficiency thinking, of waiting for the right tool that would make creative work more efficient; take the efficiency hit and just get to work!
*On [[XKCD's Money|http://xkcd.com/980/]], RM was tired of hearing disingenuous "for the price of X, we could have Y": example: "for the price of a B2 bomber, you could take all the kids from the 50 poorest schools in America, and give them a private school education" (or a B2 bomber!)
**Money was drawn in photoshop (though GDP partitioning + placement was assisted with some programming); RM hit the 4,000 row limit on Google Docs (>2K data points is often too much any way for a single infographic)
**("Filename which the audience will see briefly as I press the keys" - title of RM's presentation)
*The hard part [about infographic design] is not the graphic design or the technical aspects, but finding what parts of the data you want to show, which variables are interesting / important.
**[[XKCD's Angular Size|http://xkcd.com/1276/]] (of space objects' projection on to Earth): "there wasn't a point behind this栉 don't know what's cool about the data [when I start]"
**The viewer should be able to answer "what am I seeing?" without a legend (but sometimes a legend is helpful). Example: [[XKCD's Congress|http://xkcd.com/1127/]] (this one required some programming to draw the skeleton, RM worked with a graphic designer to draw the colour overlay); the obvious growth of the far right > 1980 gave him pause (he discovered this during design)
*Sometimes simple charts (such as bar charts) can be augmented to give context and a sense of scale: examples include [[XKCD's Ice Sheets|http://xkcd.com/1225/]] and [[XKCD's Gravity Wells|http://xkcd.com/681/]]
*Good infographics make you involuntarily smarter, not more confused.
*It's OK to have stupid questions; equip people to make their own connections. example: [[XKCD's Questions|http://xkcd.com/1256/]]
*The perpetual struggle between learning new tools in order to be more efficient and doing the job but being less efficient (striking a balance, getting enough sleep). RM has started to use [[matplotlib|http://matplotlib.org/]] in recent work, such as in [[XKCD's Dominant Players|http://xkcd.com/1392/]] and in [[XKCD's Days of the Week|http://xkcd.com/930/]] (Google wildcard search on weekdays)
*[[XKCD's Land Mammals|http://xkcd.com/1338/]] has been used in academic communication
*The power of graphics to communicate probability and uncertainty, such as in [[XKCD's Lanes|http://xkcd.com/931/]] (cancer prognosis)
*[[@dragonc's notes|https://www.evernote.com/shard/s6/sh/e53ba90a-0dac-4e2b-a269-c47fe666fc3b/bd20451554a679a4078288c8b7b098a1]]
!!Edward Tufte (ET)
>//Edward Tufte will talk about his new work, The Thinking Eye.//
[[Thinking Eye handout|https://dl.dropboxusercontent.com/u/6397998/STDP-08.07.14/Tufte-ThinkingEye-08.07.14.pdf]] (0.5 MB PDF)
*"High science and high art demand seeing"
*[[Paul Klee's Thinking Eye (notebooks)|http://www.amazon.com/Paul-Klee-thinking-Documents-modern/dp/B0007DO09W]]
*//When I cannot see I cannot understand// - Feynman
*Paul Klee's [[Fish Magic|http://www.wikiart.org/en/paul-klee/fish-magic-1925]] and Klee's attempts to see the elements of art/painting
*Galileo had the first 'Big Data', as documented in his //Starry Messenger//, with drawings of stars spilling over the margins of the page to illustrate endlessness: visible certainty wins over armchair philosophy
*See, think, design, produce: this shouldn't be interpreted as an ordered list, which would lead one to muddle through these activities
**Also: perform and reason
*Tim ~Berners-Lee's //Information Management, A Proposal// (CERN technical report, 1989) received "vague, but interesting" comment from supervisor at CERN. TBL proposed a flatness of verbs instead of a hierarchy of nouns (the problem with trees: they don't model the real world).
*Conway: // 4 groups working on a compiler design will result in a 4-pass compiler//
*University website splash pages seldom provide information you want (see [[XKCD's University Websites|http://xkcd.com/773/]]); same with government websites (ET appointed by Obama to Economic Recovery Independent Advisory Panel, it was previously very difficult to navigate to tax return information on US treasury website)
*See verbs and interactions (but not causality thinking), the spaces between and within nouns; nouns are less important
*Design requires analytical thinking (not style, nor what marketing thinks)
**Design requires to "see with [fresh/vacation] eyes" (but not an empty head); //once it becomes a word cloud, it's pretty much all over//
*Rage to produce, don't rage to conclude
*Seek serenity for the thinking eye, creative workers should beware time vampires (NYT, Twitter, FB)
*"//To prevent the legs, as they tire, from interfering with the mind// - Paul Klee on chairs
*//Shut up and look//: viewing graces trump social graces (ET's philosophy for guests viewing [[his art|http://www.edwardtufte.com/tufte/sculpture]], such as [[Megaliths, Continuous and Silent, Structures of Unknown Significance|http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0003uV]]
*"//The meaning of life is seeing//" - John Gray
*"//In ghostlier demarcations, keener sounds//" - Wallace Stevens
*On Relevance:
**speaking to those who work in other domains, ask prompting questions: "why is that?", "how do you know that?", drive them towards your interest
**"looting": to get something so specific that it becomes something other than itself
*Recall that 80% of published studies are false, and that social science is not rocket science; it is much harder than rocket science
*On Excellence:
**Learn how to know it, to get a taste for it
**How to avoid bias: "//how to review a bad book by a good friend//" - Robert Morton
**Excellence is nog normal (see citation counts of scientific articles)
**Tukey on excellence: (1) something is not expected to be useful due to oversimplification; (2) that thing is often useful; (3) it cannot be predicted when the thing works and when it doesn't
**Google Maps is a case where it shouldn't work, but it does!
**Beware chartjunk, //Tableau Turkeys//; [[Image Quilts|http://imagequilts.com/]] as a response to chartjunk and SEO in Google Image Search
**Excellence in data visualization could draw inspiration from [[Swiss mountain maps moving in time|http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0003vq]]
*Understand the relation between Evidence and Inference, esp. in fields that deal with multiple scales / domains, such as epidemiology, from microbes to diseases, to epidemics, to government policy (again, social science is harder than rocket science)
**[[The Phillips Curve|http://en.wikipedia.org/wiki/Phillips_curve]] and stagflation, explanations reaching from economics to theology
**Beware [[Big Data Hubris|http://blogs.iq.harvard.edu/netgov/The%20Parable%20of%20Google%20Flu%20(WP-Final).pdf]] (example: Google Flu Trends). //Google continues to turn hubris into renewable energy// - ET
**//if you torture the data long enough, it will confess to anything//
*Understand the survival bias (most medieval castles were built out of wood, not stone)
*[[@dragonc's notes|https://www.evernote.com/shard/s6/sh/0da072b7-f41b-4662-8679-4773db8a1d9a/ed6e4868ecc73c4592b17eb3cea7de3f]]
A theory of analytical reasoning involving a cycle of processes (while there is notion of a higher-order cycle, individual processes often come out of order, repeat, and vary considerably in the length of time required to complete - i.e. each process has an associated cost):
*Information foraging/gathering
*Re-representation of relevant information
*Manipulation of new representations to develop insight
*Communication of insights generated (presentation, decisions)
Klein (2004)'s data/frame-based theory posits that a frame is the mental structure that places organization on data, and [[Sense-making]] is the process of fitting information into that frame.
Source: <<cite Pirolli2005 bibliography:Bibliography>>, <<cite Thomas2005>>, Klein (2004)
To get started with this blank [[TiddlyWiki]], you'll need to modify the following tiddlers:
* [[SiteTitle]] & [[SiteSubtitle]]: The title and subtitle of the site, as shown above (after saving, they will also appear in the browser title bar)
* [[MainMenu]]: The menu (usually on the left)
* [[DefaultTiddlers]]: Contains the names of the tiddlers that you want to appear when the TiddlyWiki is opened
You'll also need to enter your username for signing your edits: <<option txtUserName>>
<<search>><<closeAll>><<permaview>><<newTiddler>><<newJournal "DD MMM YYYY" "journal">><<saveChanges>><<slider chkSliderOptionsPanel OptionsPanel "options 寴; "Change TiddlyWiki advanced options">>
matt's nonlinear research journal
Bureau des Services sans Spécificité
An evaluative study or experiment helps us to gauge the efficacy of a visualization tool or technique.
Novel evaluation methodologies and methods are emerging to overcome the challenges associated with studying user processes.
!!!The Overview Project
In recent years, many large corpuses of emails, reports, and other documents have been "dumped", "leaked", released, or declassified by corporations, government agencies, and other organizations. A well-known example is that of WikiLeaks, an organization that released nearly 400,000 documents relating to the conduct of armed forces and security contractors during the recent war in Iraq. Since that time, journalists have [[reported|http://goo.gl/qqGGh]] on what was contained in this corpus, which included startling patterns of civilian casualties, friendly fire incidents, and observed breaches of protocol. My goal has been to better understand how journalists and researchers explore or "mine" these document corpuses, how they seek information to support or refute prior evidence, or how they come to discover unanticipated stories hiding in the data.
Areas of specialization and experience among journalists and humanities researchers are changing, reflecting the shift toward online content presentation and the necessity to address the growing amount of structured and unstructured information at one's disposal. As a result, it is difficult to predict how and when a data visualization tool will be used, who will be using it, and whether it will be an effective part of the process of writing a convincing news story or research paper. These were the motivating questions of our collaborator at the Associated Press, who recently worked with our group to develop Overview, a visualization tool to support the process of mining large document corpuses (<<cite Ingram2012 bibliography:Bibliography>>). A prototype of [[Overview|http://www.overview.ap.org]] was released publicly in early 2012. Overview's user interface, shown in the accompanying figure, is comprised of set of linked views for exploring hierarchical clusters of related documents within a corpus, providing means for reading documents, as well as for tagging them with personally meaningful terms or phrases.
[>img(50%, )[Figure 1. A figure produced using Overview, displaying Wade�l corpus: (a) a topic tree displays hierarchical clusters of related documents, (b) document tagging functions and a list of user-generated tags, (c) a scatterplot of documents in the corpus; nearby documents are similar, (d) a list of documents having the currently selected tag in (b), (e) an embedded document viewer displaying the currently selected document in (d).|http://cs.ubc.ca/~brehmer/research/rpe/overview_jw.png]]
The need to mine large document corpuses will increase in the coming months and years. Current practices are often impractical, and gravitate to either keyword searches or brute-force approaches: reading or skimming all documents in a corpus. The former approach requires one to know a priori what one is looking for, while the latter is too time-consuming, difficult to streamline and manage.
In both cases, exploratory analysis is poorly afforded: it is impossible to sample representative documents in a corpus and extract the trends or patterns. Overview has been designed to make this exploration possible.
My intent of conducting a post-deployment evaluation of Overview has been to assess whether or not it meets the exploratory data analysis needs of target users, individuals with large document corpuses and hunches about potential stories contained within them; could Overview make writing a news story or research article possible in situations where doing so was previously impossible, or at least highly impractical?
An ideal research methodology for this work would include in-situ interview and observation sessions, in the spirit of longitudinal insight-based evaluations (<<cite Saraiya2006>>) and multi-dimensional long-term case studies (<<cite Shneiderman2006>>).
My primary data collection method is that of an in-depth, open-ended interview, recorded for later transcription.
Multiple interviews with each participant would be ideal, as processes change over the course of an investigation.
I compliment interviews by gathering texts and other information from participants, such as Overview's log file of timestamped interface interactions. I also gather information regarding their data, such as the format and number of documents contained in the corpus under investigation. Finally, I request copies of notes participants take during the course of their investigation.
To date, two professional journalists and a pair of academic researchers have completed an analysis of a large document corpus using Overview. I am also aware of several additional journalists and academic researchers who may be currently using it. Finally, I am aware of a journalist, an academic researcher, and business consultant who abandoned use of Overview, as it either did not meet their needs or was incompatible with their existing workflow or set of tools.
The discovery of prospective users from fields outside of journalism was unanticipated, indicating that Overview may support exploratory data analysis in the digital humanities, communications, and related domains.
Of the two journalists who completed an analysis of a document corpus using Overview, one published his [[findings|http://tulsaworld.com/tpdtechemails]].
Beginning with an anonymous tip and hunch relating to a botched, $4 million police equipment purchase, JW accumulated a document corpus of 6,000 Tulsa Police Department emails via a municipal records request. Using Overview, the journalist discovered newsworthy evidence contained in only a handful of emails: several police officials were responsible for the poorly managed purchase, and were caught in a potential conflict of interest with an equipment supplier.
All things considered, could JW have carried out his investigation without Overview? He admitted the possibility, however it would have taken an estimated four months of full-time dedication to read the entire corpus, while maintaining the same level of organization that Overview provided him. By contrast, his investigation using Overview took less than two weeks. With news agency deadlines and article quotas to consider, a longer-term project would have been relegated to a part-time assignment; upon its completion, the story would have run the risk of no longer being newsworthy.
!References
<<bibliography>>)
*Administrivia
**Meetings w/ TM (Thu 10:00)
***Agenda sent beforehand
**Meetings w/ JM TBD (next week)
**Office/Equipment
*~InfoVis group meetings, schedule & format
**today 15:00: to read TM textbook ch. 8 for review / discussion
**Nov 3, Dec 15 meeting discussion lead spots
***Potential paper discussion: <<cite Lam2011 bibliography:Bibliography>>
*Just read: <<cite Lam2010>>
*To read:
**<<cite Carpendale2008>>
**BELIV'10 proceedings
**BELIV'06 (+ Bertini's list)
**<<cite Saraiya2004>>
**<<cite Amar2004a>>
**<<cite Gonzalez2003>>
**TM textbook ch. 3 Cockburn references
***<<cite Cockburn2000 bibliography:Bibliography-ToRead>>, <<cite Cockburn2001>>
**BELIV'08 proceedings
*To do:
**Email TM for Pirolli book (Information Foraging Theory)
*One-off meeting w/ JM Oct 13 1:30-2pm
**Regular meetings TBA
**Involvement /consulting with ~C-TOC project (M.Sc research)
**reviewing JM's discovery grant application
*Qualitative methods courses
**EPSE 595 ([[syllabus|http://web.mac.com/sandra.mathison/teaching/courses_files/EPSE%20595%20Qualitative%20Research.pdf]]): qualitative research methods course (winter 2012) (no feedback yet from mux-grads)
**alternative #1: SOCI 503 ([[syllabus|http://www.soci.ubc.ca/fileadmin/template/main/images/departments/anso/files/grad/Soci_Grad_Outlines/2010W_Currie_503.001T2.pdf]]) (JD registered in this course Winter 2012)
**alternative #2: reading relevant texts as needed.
**''decision'': sit in on courses, wait and see
*[[Literature Review]] / [[Reading|References]]:
**Read this past week:
***[[Information Visualization Evaluation: Meta-Analysis|]]
****<<cite Lam2011 bibliography:Bibliography>> (~InfoVis group meeting discussion Oct 13)
****<<cite Carpendale2008>>
*** [[Information Visualization Evaluation: Qualitative Methods]]
****<<cite Saraiya2004>>
****<<cite Shneiderman2006>>
****<<cite Amar2004a>>
****<<cite Gonzalez2003>>
***TM textbook ch. 8 (~InfoVis group meeting Oct 6)
**To read (priority):
***BELIV'10 proceedings, Information Visualization Journal (requires [[SAGE pub access|http://ivi.sagepub.com/content/current]], delayed 12 months by UBC library)
****<<cite Scholtz2010>>
****<<cite Sedlmair2011>>
***<<cite Ellis2006>>
***<<cite Trafton2000>>
***<<cite Andrews2008>>
***<<cite Pirolli2005>>
**To read (low priority):
***<<cite Pirolli2009 bibliography:Bibliography-ToReadPriority>>
***<<cite Wattenberg2005 bibliography:Bibliography-ToRead>>
***<<cite Cockburn2001>>, <<cite Cockburn2000>>
*Research wiki development:
**[[Journal]]
**[[Meeting Minutes]]
**[[Glossary]]
**[[References]]
*Task-switching and qualitative analysis
**S.Ingram's work with J. Stray - use of MDS/Glimmer in journalism (hacker journalism, i.e. wikileaks info dump)
**crisis-driven vs. curiosity-driven multi-tasking w/ vis tools
**long term planning of analysis and working partnerships
*Upcoming Grad Student Workshops and Events, [[Graduate Pathways to Success Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]
**From Academic Researcher to Commercial Writer (2 day workshop, Oct 19-20)
**Project mgmt. workshops Oct 25 2pm (2 hour intro), Dec 6 (all day)
**Scientific/technical writing workshops Dec 1, Dec 8
*Next meeting: Oct 20 5-6pm
Links in bold blue open on the same page.
*Meeting 5-6pm
*[[Literature Review]] / [[Reading|References]]:
**[[References: Read]]:
***[[Information Visualization Evaluation: Meta-Analysis|]]
****<<cite Andrews2008 bibliography:Bibliography>> - short BELIV '08 eval. taxonomy for ~InfoVis
****<<cite Ellis2006>> - very editorial, even humorous take on eval. in ~InfoVis, and how eval doesn't always = research
****<<cite Sedlmair2011>> - //Information Visualization// journal paper, extended from BELIV '10 paper, pulled from his website
****<<cite Plaisant2004>> - summary of challenges of ~InfoVis eval
*** [[Information Visualization Evaluation: Qualitative Methods]]
****<<cite Isenberg2008>> - grounded evaluation paper from BELIV '08
****<<cite Trafton2000>> - cognitive task analysis / protocol analysis paper
****<<cite Seo2006>> - case study and email survey methodology
****<<cite Scholtz2010>> - //Information Visualization// journal paper unavailable at UBC, read BELIV '10 paper - meta-analysis of VAST reviews as design / eval heuristics
****<<cite Isenberg2008a>> - observatory lab study of collaborative data analysis
****<<cite Saraiya2006>> - follow up to <<cite Saraiya2004>>, longitudinal diary study of insights
****<<cite Saraiya2010>> - //Information Visualization// journal paper unavailable at UBC, read BELIV '10 paper - comparing user studies to insight method
****<<cite Mayr2010>> - //Information Visualization// journal paper unavailable at UBC, read BELIV '10 paper - observing problem solving (I didn't like this one)
*** [[Information Foraging and Sensemaking]]
****<<cite Pirolli2005>> - framework paper; title with 'information analysts' points to the same paper than without it in the title, from //Information Analysis// conference proceedings
**[[References: To Read (Priority)]]:
***remaining BELIV'10 proceedings, esp. those in //Information Visualization// (requires [[SAGE pub access|http://ivi.sagepub.com/content/current]], delayed 12 months by UBC library)
***remaining relevant/interesting BELIV'08, '06 proceedings
***<<cite Chen2000a>> - meta-review of evaluation
***<<cite Gonzalez2003a>> - workplace study / adoption
***<<cite Perer2009>> - long-term case studies
***<<cite Valiati2006>> - BELIV '06 task taxonomy
**[[References: To Read]] (lower priority):
***<<cite Thomas2005>> - //Illuminating the Path// book from PNNL (because so many cite it, Ron recommended it)
***<<cite Pirolli2009>> - Information Foraging Theory - I aim to have an idea of whether or not this book is useful by the time of our next meeting
***<<cite McGrath1995 bibliography:Bibliography-ToReadPriority>> - I've been meaning to re-read
***<<cite Wattenberg2005 bibliography:Bibliography-ToRead>> - baby names paper
***<<cite Cockburn2001>>, <<cite Cockburn2000>> - ch.3 refs from your textbook (comparative evaluation studies)
***<<cite Lam2008>> - empirical results for meta analysis
***<<cite Amar2005>>
*Potential project ideas
**MS - design study ~EuroVis paper (see ~InfoVis group meeting slide deck)
***drawing from ~UbiComp, ~CSCW
**MS - HDD / MDS study, ethnographic study / working partnerships with research groups (local or remote)
**SI - J. Stray - use of MDS/Glimmer in journalism (hacker journalism, i.e. wikileaks info dump): [[http://jonathanstray.com/]]
***task-switching and qualitative analysis: crisis-driven vs. curiosity-driven multi-tasking w/ vis tools
***to do: read blog: http://jonathanstray.com/
**long term planning
***~VisWeek (end of March)
***~EuroVis (beginning of Dec)
*Grad Student Workshops and Events, [[Graduate Pathways to Success Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]
**From Academic Researcher to Commercial Writer (2 day workshop, Oct 19-20)
**Project mgmt. workshops Oct 25 2pm (2 hour intro), Dec 6 (all day)
**Scientific/technical writing workshops Dec 1, Dec 8
*December holiday travel plans
**departing Wed Dec 14, working remotely Dec 15-16, 19-23
**returning Wed Jan 4
**travel February 2012 TBD (approx 1 week)
*Next meeting: Nov 3 10am (next week ~VisWeek)
**contact MS for joint meeting / HDD ethnography project
*meeting w/ JM next week 11:15am Thurs
Projects:
*DR ethnography project
**Meeting w/ MS (x2) 11.11.04, 11.11.08: [[minutes|MS-11.11.04]]
***SVN/FTP setup
**Refreshing my conceptual understanding of [[Dimensionality Reduction]], reading:
***<<cite Buja2002 bibliography:Bibliography>> - visualization methodology for MDS
***<<cite Holbrey2006>> - column reduction techniques
***<<cite Munzner2011>> - textbook draft ch. 8 (attribute reduction)
***<<cite Tan2006 showAll>> - ugrad //Intro to Data Mining// textbook: appendix on DR
***[[Manifold Learning]] - Presentation by David R. Thompson for Advanced Perception.
****curve fitting less clear component of taxonomy
**Reading / taking notes on materials in project folder, in //Notes// and //Assumptions// subfolders
***[[Cognitive Work Analysis]]
***T. Moller's presentation on tasks: //~High-Dim Analysis of something-kinda-like-simulation-problems//
****simulation and DR overlap
***DR for EDA: taxonomy notes
***Summary spreadsheet
**7 of 16 user / user groups need summaries (concentrating on these, using most recent summaries as template), early summaries likely need revision
**Need info on J. Stray
*Overview case study project: text visualization for journalism
**[[3 Difficult Document-Mining Problems that Overview Wants to Solve|http://www.pbs.org/idealab/2011/10/3-difficult-document-mining-problems-that-overview-wants-to-solve297.html]] by J. Stray
**[[TheOverview Project|http://overview.ap.org/]] - they want to build [[processing|http://processing.org]] for text visualization. Lightweight, no extensive programming knowledge required. Casual use by journalists with large datasets
**TM: or DIY infovis crowd (base don recent conversation w/ JS)
**Some possible field research questions in here:
>"What are the most painful things that journalists must do with document sets? What are the things they髥 to do but can嵐y? What problems exist, and which should we focus on solving? The only way to answer these questions is to talk to users."
*[[Vismon]] project
**TBD after paper draft talk at ~InfoVis group meeting
*Project idea: UBC Text summarization research (R. Ng, G. Carenini)
Meanwhile, my ongoing [[Literature Review]] / what I've been [[Reading|References]] last week/this week:
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite vanWijk2006 bibliography:Bibliography>> (Views on Visualization)
**[[Information Visualization Evaluation: Qualitative Methods]]
***<<cite Lloyd2011>>: re-read for ~InfoVis group discussion
***<<cite Winckler2004>> - evaluation with a scenarios generated by task taxonomy
***<<cite Borkin2011>> - ~HemoVis
**[[Information Visualization: Techniques]]
***<<cite Steinberger2011>> - context preserving visual links (and commentary)
**[[Information Foraging and Sensemaking]]
***<<cite Pirolli2009>> ch. 9: Design Heuristics, Engineering Models, and Applications (web design, browsers, search engine results)
*[[References: To Read (Priority)]]
**Springmeyer (taxonomy)
**Hullman (~VisWeek) + commentary
**Bertini (~VisWeek)
**Newell & Card (psychological science)
*[[References: To Read]]
**100+ papers or book chapters
**thematically categorized
**[[References: VisWeek2011]]
**broadening the scope to other HCI communities (CSCW, Creativity & Cognition, etc.)
***pinged CT for grounded evaluation in CSCW papers (CT away on job talk)
Upcoming:
*CHI rebuttals (x2) (this weekend / next week)
*Alz. + Dementia journal paper (awaiting reviews)
*Maneesh Agrawala visit / scheduling update
**to do: find out about KB's demos
*Graduation ceremony Nov. 25
*Moving early December, exact dates TBA
*Technical writing workshop Dec 1, 8 (conflict with ~InfoVis group meetings) - I may only attend Dec 1 session
**group discussion lead Dec 8
*Working remotely Dec 14-16, 19-23 (in Ottawa) - will Skype in to ~InfoVis group meeting Dec 15
**returning Wed. Jan 4 midday
*Visiting family reading week 2012 (mid Feb - TBA)
*Getting a new Laptop
!References
<<bibliography>>
Projects:
*DR ethnography project
**Refreshing my conceptual understanding of [[Dimensionality Reduction]], reading:
***<<cite Ingram2010 bibliography:Bibliography>> - ~DimStiller
***<<cite Tenenbaum2000>> - ISOMAP
**Reading case studies / exiting summaries
***7 of 16 user / user groups need summaries (concentrating on these, using most recent summaries as template), early summaries likely need revision
***read Cindy Marven, ~GenomeDX case studies and summaries (to read: J. Wright)
***currently working on H. Lam case study
****What did reviewers say re: CHI submission?
***Need info on J. Stray
**MS sez: additional case study interviews before Dec? thoughts?
*[[TheOverview Project|http://overview.ap.org/]] - DIY info hacker crowd-source journalism use case?
*[[Vismon]] project (TBD)
[[Literature Review]]: (low priority this week due to CHI rebuttal, HDD/DR project, visitors)
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite Bertini2011>> ~VisWeek 2011 quality metrics
***<<cite Springmeyer1992>> - taxonomy of scientific data analysis tasks
*[[References: To Read (Priority)]]
**Hullman (~VisWeek) visual difficulties + commentary
**Newell & Card (psychological science, KB's suggestion)
*[[References: To Read]]
**[[References: VisWeek2011]]
**broadening the scope to other HCI communities (CSCW, Creativity & Cognition, etc.)
***pinged CT for grounded evaluation in CSCW papers
Ongoing:
*CHI rebuttal writing - good reviews!
*Mike Terry & Maneesh Agrawala visits
*~GenomeDX SPLOM vs. 3DSP
Upcoming:
*Graduation ceremony Nov. 25
*Moving early December, exact dates TBA
*Technical writing workshop Dec 1, 8 - I may only attend Dec 1 session
*Working remotely Dec 14-16, 19-23 (in Ottawa) - will Skype in to ~InfoVis group meeting Dec 15
**returning Wed. Jan 4 midday
*Visiting family reading week 2012 (mid Feb - TBA)
*laptop: Pages/Keynote/Numbers?
!References
<<bibliography>>
Projects:
*DR ethnography project
**Case studies
***5 of 16 user / user groups need summaries + early summaries likely need revision
***completed: H. Lam, D. Higgins
****vocabulary / technical details of ~Bio-DR
***Kerem. A (UBC PDF): classifying human motion - Interview 11.11.23
****Uses PCA, SFFS for visual cluster separation, classification algorithm input
***Need info on J. Stray
*[[The Overview Project|http://overview.ap.org/]] (TBD)
*[[Vismon]] project (TBD)
*Interruptions and Multitasking + ~InfoVis/VA (Decision making, collaboration)
[[Literature Review]]:
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite Hullman2011a bibliography:Bibliography>> ~VisWeek 2011 quality metrics + S. Few commentary
****<<cite Chang2010>> ~Learning-Based Evaluation (BELIV '10)
*[[References: To Read (Priority)]]
**Newell & Card (psychological science, KB's suggestion)
*[[References: To Read]]
**[[References: VisWeek2011]]
**Creativity & Cognition proc.
***CSCW (pinged CT for grounded evaluation papers)
Recent:
*M. Agrawala visit + DLS
**Video puppetry + WC's ~NapkinVis for vis. prototyping?
*MAGIC demo day 2010
**VA tools, etc.
Upcoming:
*MS SPLOM paper draft for ~InfoVis meeting
*Graduation ceremony tomorrow, Nov. 25
*Moving early December, exact dates TBA
*Technical writing workshop Dec 1, 8 - I may only attend Dec 1 session
*Working remotely Dec 14-16, 19-23 (in Ottawa) - will Skype in to ~InfoVis group meeting Dec 15
**returning Wed. Jan 4 midday
*Visiting family reading week 2012 (mid Feb - TBA)
!References
<<bibliography>>
10am Pacific, 1pm Eastern (over Skype)
Projects:
*DR ethnography project update: case studies
**18 user / user groups
**completed or draft: H. Lam, D. Higgins, K. Altun, S. ~Nabi-Abdolyousefi, J. D. Westbrook
***TM: feedback on J. Westbrook's summary?
**currently working on: A. Saad (Torsten's ~PhD student), J. Buettgen, (received translation of interview notes from latter Dec 14.)
**MB to do: revising early summaries by MS
**TM: need info on J. Stray
*[[Vismon]] project - timeline? any feedback from Torsten?
*[[The Overview Project|http://overview.ap.org/]] (TBD)
*other project areas of interest:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
**graphical perception
[[References: Read]]:
*<<cite Gigerenzer2007 bibliography:Bibliography>> - gut feelings, hunches, intuition: the intelligence of the unconscious (book)
*[[Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Chang2009>> - characterizing insight as both an event and a substance
*[[Information Visualization Evaluation: Qualitative Methods]]
**<<cite Kang2011>> - grounded eval. of intelligence analysis process ([[VisWeek|References: VisWeek2011]])
[[References: To Read]]:
*[[References: To Read (Priority)]]
*[[References: VisWeek2011]]
*Creativity & Cognition proc.
*CSCW proc.
*Charmaz, K. - [[Constructing Grounded Theory|http://www.amazon.ca/Constructing-Grounded-Theory-Practical-Qualitative/dp/0761973532]] - on 2hr reserve at UBC library - to buy a copy?
*EPSE 595 - qualitative methods readings (syllabus sent by instructor Dec 13)
This week / upcoming:
*finished moving!
*CHI paper accepted! :D - some revisions needed for camera-ready Jan. 13
*MB Air set up, iWork delivered, installed, SVN set up, VPN issue resolved
*attended CPSC 533 project presentations
*attended seminar on computational population biology: Tanya ~Berger-Wolf (U. Illinois, Chicago): [[Computational Insights into Population Biology|http://compbio.cs.uic.edu/]]
*Working remotely Dec 14-16, 19-23 (in Ottawa)
**returning Wed. Jan 4 midday
*schedule next term (emailed JM Dec 14)
*courses / breadth requirement
**asked JP for confirmation re: breadth form (it has been approved - only a Systems breadth course remains - CPSC 508 / 538W)
*Visiting family reading week 2012 (mid Feb - TBA)
!References
<<bibliography>>
4pm Wednesday
*Meeting scheduling for the term: week 1: flex/no meeting, week 2: all, week 3: TM, week 4: JM
**No regular meeting tomorrow (attending workshop)
Projects:
*[[Vismon]] project
**discussing [[Vismon: Research Questions]]
*[[HDD-DR Ethnographic Project]]
**completed summaries for each case study
**existing taxonomy summary table filled in
**reading:
***email thread re: classes and clusters
***http://thesocialcanvas.blogspot.com/2011/03/classification-vs-clustering.html
***http://www.coli.uni-saarland.de/~crocker/Teaching/Connectionist/lecture13_4up.pdf
***<<cite Rubinov2006 bibliography:Bibliography>> - Classes and clusters in data analysis
**this week: additional taxonomy brainstorming following Monday's discussion: will meet w/ MS Friday 10am
***reframing the Qs and As in light of Monday's discussion, trying to encapsulate the cross-cutting aspects into examinations of dimensions and clusters
**next week: case study write-ups?
*Other potential projects:
**[[Overview Project|http://overview.ap.org/]]
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**epistemologies: objectivism vs. constructionism vs. subjectivism
***positivism, interpretivism (symbolic interactionism, phenomenology, hermeneutics), critical inquiry
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[Reading recently|References: Read]]:
*<<cite Crotty1998>>: foundations of social research (EPSE 595 text)
*<<cite Gigerenzer2007>> - gut feelings, hunches, intuition: the intelligence of the unconscious, decision-making under uncertainty
[[References: To Read]]:
*Creativity & Cognition, CSCW proc.
*Charmaz, K. - [[Constructing Grounded Theory|http://www.amazon.ca/Constructing-Grounded-Theory-Practical-Qualitative/dp/0761973532]] - 2hr reserve at UBC library
*D. Kahneman (2011): //Thinking Fast and Slow// (addressing the same issues as <<cite Gigerenzer2007>>, however taking a different stance)
This week / upcoming:
*tomorrow: GRAND workshop at SLAIS on text analysis (Thursday AM)
*tentative plans to visit family Feb 6-14 (dependent on Vismon project)
!References
<<bibliography>>
11am Thursday
*Meeting scheduling for the term: week 1: flex/no meeting (TM), week 2: TM +JM, week 3: (TM conflict), week 4: JM
Projects:
*[[Overview Project|http://overview.ap.org/]] - data journalism - [[J. Stray|http://jonathanstray.com/]] visiting Vancouver in 12.03.05:
**logistics: itinerary for Monday - how much time do we have w/ JS? will he present to the ~InfoVis group? NICAR, data and intl' conflict symposium
***9am: discuss [[DR in the Wild|HDD-DR Ethnographic Project]] project
***9:30am - 11am and onwards: discuss [[Overview]] project
**''to do'': read recent posts:
***[[Using Overview to analyze 4500 pages of documents on security contractors in Iraq|http://overview.ap.org/blog/2012/02/private-security-contractors-in-iraq-analysis/]]
***[[What did private security contractors do in Iraq?|http://overview.ap.org/blog/2012/02/iraq-security-contractors/]]
**reviewing JS' summary for the [[DR in the wild project|HDD-DR Ethnographic Project]] (in the SVN / User Observations folder)
**~MoDisco reviews:
***R1 suggested algorithms for comparison: Paulovich et al (2008): ~Least-Square Projection (not my problem)
***R4 sez: evaluation needed, head-to-head comparison against other tools (i.e. OST*10: topic islands, Yang et al (2010): Newdle, Chen et al (2009): Examplar-based Visualization), as well as user studies (usability, utility, etc.)
>//do users qualitatively prefer ~MoDiscoTag to a state of the art system, and if so, how/why?//
>[...] //we have found that many analysts who use DR for document set visualization have the persistent unease that there is often but not always structure in their data sets that is not revealed// - evidence from [[DR in the Wild|HDD-DR Ethnographic Project]]
**[[Overview Discussion 12.03.05]]
*[[Vismon]]
**[[Vismon: Research Questions]], further response from Randall, Torsten
*[[DR in the Wild|HDD-DR Ethnographic Project]]
**validating, iterating on our Q&A taxonomy table with usage patterns from interviews and usage papers
**I have methodology token - describing what we did, received feedback from EPSE 595 prof
**meeting w/ MS tomorrow
*Other potential projects:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique, then possibly extending to HD to HDD/DR data
***to do: contact author, find out if anyone is validating the method - nothing published so far
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**[[Interviewing]] - social constructivist perspective, structured, open-ended, post-modern
**[[grounded theory|Grounded Evaluation]]
**[[Ethnography]] - MIT anthropologists studying marine biologists in the wild
**[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
**[[assignment 2|EPSE 595 Workbook 2]] participant observation, field notes
**upcoming: focus groups. found data, data analysis and representation
***[[assignment 3|EPSE 595 Workbook 3]] interviews
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[Reading recently|References: Read]]:
*<<cite Wickham2010>> - graphical inference
*<<cite Becker1957>> - participant observation vs. interviewing (for EPSE 595)
*<<cite Fontana1994>> - on the Art of Interviewing (for EPSE 595)
*<<cite Krueger2002>> - on focus groups (for EPSE 595)
[[References: To Read]]:
*Hayes' Action Research in HCI (referred to by MS)
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project)
Upcoming:
*CHI submission
**registration (Mar 13)
**CHI madness preparation (Mar 23)
*qual. methods course talk to MUX in April / May
*RPE timeline: see [[notes from previous meeting|TM-JM-12.02.16]]
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
**other ideas?
11am Thursday
*Meetings in March: 03/1: flex/no meeting (TM), 03/08: TM +JM, 03/15: (TM conflict, JM instead), ''03/22'': (JM conflict, TM instead), 03/29: (no meeting, ~InfoVis submission deadline)
Projects:
*[[DR in the Wild|HDD-DR Ethnographic Project]]
**paper draft reading Friday w/ ~InfoVis group
**refs/citations
**to do: supplemental material
*[[Overview]] Project
**[[Overview Discussion 12.03.05]]
**2 studies being run in parallel:
***[[Overview Deployment Field Survey]] - post-deployment field data collection from users - to tie in with [[EPSE 595]] final project proposal (Apr 11)
***insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
**to do:
***iterate on [[deployment survey Qs|Overview Deployment Field Survey]]
***brainstorm methodology for #2
***project timeline and the RPE, determine supervisory committee, see [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*Other potential projects:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work - research Qs addressing context of use
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique, then possibly extending to HD to HDD/DR data - to contact HW after ~InfoVis deadline (Mar 30)
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**Recent Topics:
**[[Material & Environmental Data]]
***[[Narrative Analysis|http://prezi.com/i9c0uupsydvi/narrative-analysis/]]
***[[Organizing and Making Sense of Data]]
***[[Action Research|http://prezi.com/0lc7aiia6rlx/action-research/]]
**Assignments:
***[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
***[[assignment 2|EPSE 595 Workbook 2]] participant observation, field notes
***[[assignment 3|EPSE 595 Workbook 3]] interviews
***[[assignment 4|EPSE 595 Workbook 4]] found data / material culture
**Upcoming:
***assignment 5: data analysis and representation, themes (Apr 4)
***final project: interpretive / critical project proposal: [[Overview Deployment Field Survey]] (Apr 11)
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[References: To Read]]:
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Upcoming:
*CHI submission
**registration (Mar 13) - done
**CHI madness preparation (Mar 23) - draft + feedback*qual. methods course talk to MUX in April / May
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
11am Thursday
*Meetings in April: 04/05: flex meeting (JM), 04/12: (TM +JM), ''04/19: (TM)'', 04/26: (JM - last meeting before GRAND / CHI)
Projects:
*[[Overview]] Project - see [[notes from meeting w/ JS|Overview Discussion 12.03.05]]
**2 studies being run in parallel:
***1. [[Document mining in data journalism]] - post-deployment field data collection from users - [[EPSE 595]] final project proposal (submitted Apr 11) - ''awaiting feedback''
****[[interview foci and questions|Overview Deployment Field Survey]]
****User #1: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - ''still in progress as of Apr 13''
*****followed-up w/ JS - piloting, coordinating interviewer roles, interviewing AP Caracas bureau chief -'' awaiting response''
***2. insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
****possible VIVA connections via RR to UBC journalism school, Vancouver Sun
****''to do'': brainstorm methodology, ''awaiting response from JS'' re: involvement of faculty and students
**''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
**email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
>''[a. if you have any thoughts on a better way to collect user data]'':
>A big ongoing question. We in human-computer interaction (HCI) face this problem as well, particularly outside of laboratory settings. Methods vary considerably depending on what user data we intend to collect; for example, methods to gauge user productivity might be similar to those for determining speed and error rates, but these may differ substantially from methods for quantifying insights, or those for measuring collaboration, motivation, affective response, or user engagement. Browsing the HCI literature for "in-the-wild" studies, along with whatever you're intending to measure, often turns up some interesting ideas; the [[ACM|http://dl.acm.org/]] and [[IEEE Xplore|http://ieeexplore.ieee.org/]] digital libraries are good places to start. In particular, several peer-reviewed conference proceedings are dedicated to methods for designing and evaluating technology that depend heavily on context of use, these being [[Computer-Supported Cooperative Work|http://cscw2012.org/]], [[Pervasive Computing|http://pervasiveconference.org/]], [[Ubiquitous Computing|http://www.ubicomp.org/]], and [[Persuasive Computing|http://captology.stanford.edu/]].
>
>The best thing we can hope to do 'in the wild' is triangulate between multiple collection methods, relying on both manufactured and found data. The former would include detailed interaction logs, user diaries (with email / text message reminders so that users adhere to a regular diary schedule), interviews, and online surveys. The latter would include video, audio, and screen capture recordings, and collecting artifacts, produced or encountered by users. The latter may include browsing histories, or analytical artifacts, such as visualizations, produced in the course of their day-to-day work.
>
>''[b. what you think of my research]'':
>My response to this question is largely bundled with my response to (c). It's very interesting work, especially with regards to data collection and analysis methods. I appreciate seeing individuals and processes characterized using a different methodological lens. Until recently, my research projects have been largely quantitative and empirical in the post-positivist tradition - controlled lab studies and surveys - but my current research questions have me looking to other fields for methodological insight. As of late, I've been doing a lot of reading about grounded theory, ethnography, and action research. I recently completed a graduate course in qualitative research methods, and I'm now trying to look at my research questions from an interpretive or critical viewpoint. The HCI field, still predominantly guided by the post-postivist thinking of cognitive psychology and human factors engineering, requires more rigorous instances of this type of work, so it's refreshing to see good examples of it carried out within research in other disciplines. Your methods paper using ~Actor-Network Theory is particularly interesting for this reason.
>
>I'd also add that I enjoyed reading your description of political maps, how they are both actors and objects. This complements my earlier understanding of maps, which comes from a perceptual psychology and geospatial design standpoint.
>
>''[and c. if you think there are any ways we might work, if not together, at least in a way that will allow us to help each other out. I know you're a ~PhD student, and are therefore doing a lot of long-term solo research on Overview. But I'd love to know if there's a way we might be able to assist each other, in ways that don't tread on the work each of us is doing.]'':
>I agree entirely with this last point. After reading your interview, blog post, and paper, I see our research interests as being largely complementary to one another, rather than overlapping. From what I've surmised, you're addressing big issues relating to transparency within the tangled web of modern journalism, the roles of collaboration and technology, how this type of journalism is received by fellow journalists and the news-reading public, and where news production is heading. Whereas my lens is directed toward characterizing the data analysis process within domains that are facing growing amounts of data, journalism being one. The purpose of which is ultimately to inform the design of better data analysis tools. Distributed networks of collaboration, as well as changing standards and expectations in response to big data, are undoubtedly factors that shape the data analysis process in all of these domains. Characteristics of data, existing analytical tools, and domain-specific tasks are other important factors. The Overview project is a starting point for me, it is an opportunity to characterize this process in the field or journalism. My research plan is to look for parallels and differences in the data analysis process across several domains going through this 'big data flux'. For this reason, I found the electoral politics case study in your paper very inspiring, as this is another domain I can look to in the future.
>
>That being said, I agree that it would be mutually beneficial to stay informed about each other's work. The way I see it, your ethnographic research will inform my characterizations of data analysis, as well as design guidelines for new tools. At the same time, new data analysis tools and new insight into analytical processes may reciprocally inform your work.
***response from JS re: CWA collaboration
>//This is a really awesome conversation or eavesdrop. Enjoying both your work. And the first Overview user who isn't me is grappling with it right now. Also, Dan Cohen (Mr. Digital Humanities) and friends are trying to use it on some historical texts. So lots going on.//
**software:
***JD on recording Skype calls: [[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]]
***[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595
***[[dedoose|http://www.dedoose.com/]] - J. Woefler / CT, MH recommended
**~MoDisco to become tech report, JS to blog sections of it
*Other projects:
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>), controlled lab study to validate the technique, then extend to HDD/DR data
**follow-up journal paper w/ CT on interruptions and ~C-TOC: ''to do'': outline / pre-paper talk before GRAND/CHI
Courses:
*[[EPSE 595]] - qualitative research methods
**final project: interpretive / critical project proposal: [[Document mining in data journalism]]
Reading:
*continue reading <<cite Charmaz2006>> - Grounded Theory
To Read:
*Confessions of a Grounded Theory ~PhD (Furniss, CHI 2011)
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Upcoming:
*CHI practice talk at MUX, April 25 - to do: prepare this presentation
*In Texas @CHI May 6-15
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
!!References
<<bibliography>>
11am Thursday
*Meetings in May: ''05/02: TM'', 05/10: (at CHI), 05/17: TM, 05/24: JM, 05/31: flex / as needed
[[Overview]] Project
*Exciting methodological overlap b/w the two studies: both are largely qualitative, both involve recording insight. However the two studies differ in two important ways: (1) professionals vs. students; (2) personal vs. prepared datasets (both being be real, and not toy, datasets; datasets used by Overview users in the course of their ongoing work vs. datasets we are familiar with that we give to students.
*1. [[Document mining in data journalism]] - post-deployment field data collection from users
**[[interview foci and questions|Overview Deployment Field Survey]]
**User #1: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - 'still in progress as of May 01
>''JS'': //[JW] seems to be finally in the thick of the analysis, having just really gotten Overview working on his documents (which is itself a story, I think!) In any case, you can hopefully get a flavor of where we are from the below.//
>
>[...] //Looks like he's just really getting started. Getting the data set up for analysis -- sometimes called ETL (extraction, transformation, loading) -- is recognized to be a big deal in the commercial analytics world. It's something Overview has to deal with seriously as a project but I'm not quite sure how your study will or won't address this.//
**User #2: [[Dan Cohen|http://www.dancohen.org/]], Associate Professor in the Department of History and Art History at George Mason University and the Director of the Roy Rosenzweig Center for History and New Media.
**User #3: K. Watson at ITP / NYU: [[ITP thesis archivers using Overview|http://blog.itparchive.com/post/20561066325/experimenting-with-the-associated-press-developed]]
**followed-up w/ JS - piloting, coordinating interviewer roles, interviewing AP Caracas bureau chief
***JS happy to do pilot interview
>''JS'': //We can certainly talk with the Caracas bureau chief about his experiences with Overview. However it was a very early version, had lots of interesting usability problems, and it did not have logging. So the data won't be as good there. ''But I assume you've seen the notes he wrote?''//
***got AP Caracas notes from TM - mainly usability notes;
**Advertise study ("tell us what you're able to do in Overview") on the AP site? (asking JS)
**What is Overview currently logging?
**software:
***[[Skype Premium|http://www.skype.com/intl/en-us/prices/premium?intcmp=CS-Upsell-NarrowRight]] ($60 / annually)
***JD on recording Skype calls: [[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]] (~20$)
***[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595 ($39 education license)
***[[dedoose|http://www.dedoose.com/]] - J. Woefler / CT, MH recommended (web app, $13 monthly subscription service, 15% discount if bought for 6mo/12mo), has transcription functionality
*2. [[An Insight-Based Evaluation of Overview]] with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
**possible VIVA connections via RR to UBC journalism school, Vancouver Sun, followed-up on Friday, ''awaiting response''
**[[open brainstorming questions:|An Insight-Based Evaluation of Overview]]
***Between or Within Subjects?
***Longitudinal ~In-The-Wild vs. ~Single-Session ~In-The-Lab?
***Open coding vs. theoretical coding? (potentially another paper's worth of material here)
***Video screen-sharing show-and-tell / insight walkthrough
*''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*compiling [[Data Journalism: Links and Literature]] (from JS, CWA)
**[[The Data Journalism Handbook|http://shop.oreilly.com/product/0636920025603.do?sortby=publicationDate#]] by Jonathan Gray, Lucy Chambers, Liliana Bounegru (O'Reilly, 2012)
*email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
[[Graphical Inference User Studies]] Project
*[[compiled list of papers|Graphical Inference Evaluation]] citing original graphical inference paper, 2 papers conducting a graphical inference user evaluation:
**Wood et al (2011) - ~BallotMaps (TVCG)
**Kairam, Heer et al (2012) - ~GraphPrism (AVI)
*configuring [[Ggplot2|http://had.co.nz/ggplot2/]] for R, [[nullabor|https://github.com/ggobi/nullabor]] package for graphical inference
*''to do'': read Heer, Kosara, et al Mechanical Turk papers
[[EPSE 595]]
*[[final research proposal|Document mining in data journalism]] ([[Document mining in data journalism]])
>@@color:#444bbb; ''SM'': //"All sounds good. is there any sort of tracking embedded in Overview? If so, you might also be able to see how journalists move within and between documents when data mining. This may be beyond what you want to do, but studying the pathways through data and documents might reveal something about decision-making, choices and conclusions (as evidenced in what the journalist ends up writing)."// @@
>
>@@color:#444bbb; ''SM'': //"what a pleasure to read. Clear, well reasoned and absolutely doable. Good luck with the research."//@@
[[recently read|References: Read]]:
*<<cite Furniss2011 bibliography:Bibliography>> - confessions of a [[Grounded Theory]] ~PhD (CHI)
*<<cite vanderMaaten2011>> - multiple maps t-SNE for ~InfoVis group meeting
*<<cite Shipman1999>> - formality considered harmful (cited by Semantic Interaction: <<cite Endert2012>>)
*<<cite O'Brien2010>> - insight-based evaluation for Gremlin: [[notes|Information Visualization Evaluation: Qualitative Methods]]
*<<cite North2011>> - Insight evaluation IV journal paper - comparison with benchmark task method
*<<cite Charmaz2006>> - Grounded Theory ref (''ongoing'')
Upcoming:
*CHI May 6-15 (practicing / rehearsing / memorizing ongoing...)
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
!!References
<<bibliography>>
11am Thursday
*Meetings in May: 05/02: TM, 05/10: (at CHI), ''05/17: TM'', 05/24: JM, 05/31: flex / as needed
[[Overview]] Project
*Exciting methodological overlap b/w the two studies: both are largely qualitative, both involve recording insight. However the two studies differ in two important ways: (1) professionals vs. students; (2) personal vs. prepared datasets (both being be real, and not toy, datasets; datasets used by Overview users in the course of their ongoing work vs. datasets we are familiar with that we give to students.
>@@color:#444bbb; ''JS'':Well the obvious choices for me might be Columbia, or another j-school in NY. Looks like I'm going to be co-teaching a course at Columbia in the fall, so that's probably most likely.@@
>
>@@color:#444bbb;Of the handful of document sets that someone has really looked at in Overview, it seems to take 6-12 to get a good set of tags for document sets in the 10k range. Perhaps we could find ways to reduce that, but it definitely seems like something that would have to be done for course credit.@@
>
>@@color:#444bbb;But, how does this relate to collecting information about use "in the wild" ala Jarrel Wade? Do we still do the same sort of interview protocol for the students vs. professional users? Are there two different study designs here?@@
>
>''MB'': there would be two study designs, however they would be similar in many respects. Some key differences:
>
>"in the wild" users ala Jarrel Wade would use Overview with their own data, while we would provide the dataset for students to work with.
>students would get both study conditions, Overview and search-only. We will rely on retrospective anecdotes from "in the wild" Overview users about prior search-only projects (if any).
>students' time would need to be constrained in order to align with their academic calendar, while we don't have as much control over how long "in the wild" users spend with Overview, as many may be working with deadlines
>
>The two studies would be similar in terms of data collection and analysis methods. In both cases we will conduct interviews with users and collect artifacts (log files, screen captures), and the analysis of this data will be similar for both studies, facilitating comparisons between students and "in-the-wild" Overview users.
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[interview foci and questions|Overview Deployment Field Survey]]
*User #1: JW: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - 'still in progress as of May 07
>@@color:#444bbb; ''JS'':He's not quite done yet but in the home stretch, and I think there's some really interesting stuff in here.@@ (JW email)
*User #2: [[Dan Cohen|http://www.dancohen.org/]], Associate Professor in the Department of History and Art History at George Mason University and the Director of the Roy Rosenzweig Center for History and New Media.
**Fred Gibbs: email on Overview in Digital Humanities (May 4/8)
*User #3: K. Watson at ITP / NYU: [[ITP thesis archivers using Overview|http://blog.itparchive.com/post/20561066325/experimenting-with-the-associated-press-developed]]
*followed-up w/ JS - piloting, coordinating interviewer roles, interviewing AP Caracas bureau chief
**pilot interview later this week / next week
**I take ownership on questions
**AP Caracas notes - mainly usability notes
*Advertise study ("tell us what you're able to do in Overview") on the AP site (done)
*Overview is currently logging (have JS' log file, from war logs example):
**program start and shutdown
**tag creation and deletion
**adding and removing documents from tags
**view document in integrated viewer
**view document in external browser
**MDS view select, pan, zoom
**load tag file
**select node in tree view
**change tree pruning level
2. [[An Insight-Based Evaluation of Overview]] with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
*Between or Within Subjects?
>@@color:#444bbb; ''JS'': Why do you want to have each subject try both methods?@@
>
>''MB'': A few reasons. First, it's a tradeoff: a within-subjects study would require less participants but more time per participant. It will allow us to ask all participants to make comparisons between both methods; we'll be able to study whether the ordering of methods has an effect on the resulting analysis. It will also allow us to better detect particularly strong and weak students (regardless of method).
**Longitudinal ~In-The-Wild vs. ~Single-Session ~In-The-Lab?
>@@color:#444bbb; ''JS'': Sounds like little bit of a different protocol than the think-aloud approach used by previous researchers. Depends on the subjects to understand what an "insight" is and keep good notes, no? Or perhaps from the walkthrough and the tagfile/logs we could reconstruct the time of each insight?@@
>
>@@color:#444bbb;And yes, I think looking at the final stories, if any, is really interesting. It's relevant to a possibly different set of questions, regarding how much of the insight makes it into final product.@@
>
>''MB'': Due to the longitudinal aspect of this study, we won't be able to use a think-aloud approach. Instead, we'll rely on a guided retrospective walkthrough of their analysis via video chat. Re: keeping notes and understanding what constitutes insight, we can provide participants with some initial prompts or characteristics of insight. A few of the published insight studies have asked participants to list potential findings, hunches, or questions about the data before exploring a dataset; whenever an item on this list is addressed during their analysis, they would be encouraged to take note. Maintaining regular notes could also be a requirement of the course assignment.
**Open coding vs. theoretical coding? (i.e. the Amar / Stasko taxonomy)
>potentially another paper's worth of material here
**Video screen-sharing show-and-tell / insight walkthrough
Overview misc:
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: http://lensingwikipedia.cs.sfu.ca/
*''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*possible VIVA connections via RR to UBC journalism school, Vancouver Sun, followed-up, ''awaiting response''
*compiling [[Data Journalism: Links and Literature]] (from JS, CWA)
*email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
*software:
**[[Skype Premium|http://www.skype.com/intl/en-us/prices/premium?intcmp=CS-Upsell-NarrowRight]] ($60 / annually)
**[[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]] (~20$)
**[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595 ($39 education license)
**[[dedoose|http://www.dedoose.com/]] - JW, CT, MH recommended (web app, $13 monthly subscription service, 15% discount if bought for 6mo/12mo), has transcription functionality
[[Graphical Inference User Studies]] Project
*[[compiled list of papers|Graphical Inference Evaluation]] citing original graphical inference paper
*''to do'': read Heer, Kosara, et al Mechanical Turk papers, some CHI papers re: crowd sourcing
[[recently read|References: Read]]:
*<<cite Krzywinski2011 bibliography:Bibliography>> - Hive Plots for ~InfoVis group meeting
*<<cite Charmaz2006>> - Grounded Theory ref (''ongoing'')
[[to read|References: To Read]]:
*Heer, J. and Shneiderman, B. (2012). Interactive Dynamics for Visual Analysis. Communications of the ACM.
*[[References: CHI2012]]
Upcoming:
*IMAGER social WIP presentation (May 23)
*UBC awards annual progress report (May 31)
!!References
<<bibliography>>
11am Thursday
*Meetings in June: ''06/07: TM'', 06/14: TM+JM, 06/21: TM+JM, 06/28: TM+JM
[[Overview]] Project
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[interview foci and questions|Overview Deployment Field Survey]]
*User #1: JW: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled - 'still in progress as of May 07
**[[JW's story|http://tulsaworld.com/tpdtechemails]]
**[[project on DocumentCloud|http://www.documentcloud.org/public/search/projectid:%205269-tpd_emails]]
**JW sent overview_log, tag file, source data CSV, loading into Overview, excellent notes
**49 reader comments, some relevant to analysis methods used by JW
**1h40min interview yesterday via G+ hangout, screen sharing, recorded video and audio - will begin transcription today
**follow-up via email, expecting consent form to be returned
*other users' status? - keeping track in ~GDoc / Dedoose
*ask JM re: ethics, changing names and titles on consent forms.
*software:
**[[Skype Premium|http://www.skype.com/intl/en-us/prices/premium?intcmp=CS-Upsell-NarrowRight]] ($60 / annually) - audio quality drops for screen-sharing presenter during group video chats (reported problem on Skype user forums)
***[[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]] (~20$) - Skype-add on, cannot be used for other apps
**G+ hangout video chat + screen sharing
**[[IShowUHD|http://www.shinywhitebox.com/ishowu-hd/]] screen video + audio capture
**[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595 ($39 education license) - might have better video transcription features than Dedoose
**[[dedoose|http://www.dedoose.com/]] - JW, CT, MH recommended (web app, $13 monthly subscription service, 15% discount if bought for 6mo/12mo, $11 / per user for 2+ users), has transcription functionality (limited); $0.25 - $0.75 per hour of video per month surcharge
**expensesۛAn Insight-Based Evaluation of Overview]] with data journalism students (Columbia: JS has part-time instructor post this fall)
*discussion w/ JM last week - UBC ethics for this project TBD when JS's teaching post / ability to conduct study confirmed
Using Overview (and other document mining tools, techniques, scripts):
* [[Visually exploring 13,000 music reviews with Overview|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]] - JS says this is the largest dataset used so far in Overview
*Overview w/ Vancouver city council meeting minutes (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]) (~2,500 pages)
*~2,000 news articles from various media outlets relating to Vancover's DTES (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]])
Overview misc:
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: http://lensingwikipedia.cs.sfu.ca/
**TMailable: Jun 3-15; AS unavailable May 27 - Jun 7; Jun 20 - Jul 21 (window: Jun 16-19; May 23-25)
*''RPE'': RR joins committee [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*possible VIVA connections via RR to UBC journalism school, Vancouver Sun
**email thread w/ K. Melnick, VIVA VA challenge group member re: Overview - likely to give a demo to big group at some point, discussed Vancouver Sun collaboration
***also collaborating w/ Anoop
**met K. Rozendal, Vancouver Sun reporter covering Saul's visit, interested in Overview; also a UBC ~J-School student - interested in using Overview with Vancouver city council meeting minutes
*Skype call yesterday w/ F. ~McKelvey, communications ~PhD candidate at Ryerson, PDF at U of Washington this fall - big data, network analysis, elections data, also reading C. W. Anderson
*compiling [[Data Journalism: Links and Literature]] (from JS, CWA)
*email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer - ''awaiting response''
[[Graphical Inference User Studies]] Project
*[[compiled list of papers|Graphical Inference Evaluation]] citing original graphical inference paper
*read <<cite Willett2012 bibliography:Bibliography>>
*''to do'': read Heer, Kosara, et al Mechanical Turk papers
[[recently read|References: Read]]:
*<<cite Jianu2012>> - small UI changes affect analytic strategies (CHI 2012): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*<<cite Heer2012>> - Colour naming for ~InfoVis group meeting
*<<cite Brandes2012>> - Gestaltlines for ~InfoVis group meeting
*<<cite Eisenstein2012>> - (2012 CHI ~TopicViz WIP): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*<<cite Charmaz2006>> - Grounded Theory ref (''ongoing'')
[[to read|References: To Read]]:
*Heer, J. and Shneiderman, B. (2012). Interactive Dynamics for Visual Analysis. Communications of the ACM.
*[[References: CHI2012]]
Upcoming / ongoing:
*DRITW rejected from ~InfoVis - fast tracked to TVCG - next steps?
*~ToCHI journal paper on domestic interruptions with CT, JM
*vacation middle week of July / beginning of August
!!References
<<bibliography>>
*Meetings in Sep: ''TM: 09/05'', TM: 09/12, JM: 09/19, TM/TBD: 09/26
[[Overview]] Project
*RPE paper draft v3
**addressed minor edits, figure revisions
**revised intro framing, added refs
**supplemental materials: annotated bibliography, EPSE 595 proposal, question list, presentation for JS (includes initial GT code list)
**what did I do?: active voice / more method execution details
**remaining to be done:
***subtler distinction b/w exhaustive search vs. exploratory search
***reflective para in Discussion re: findings in light of larger ~PhD goals
*RPE presentation scheduling:
**presentation details in grad handbook
**TM/JM to arrange a date/time with RR and recruit a chair from the GAC (dept. invited to presentation)
**MB to send out report to TM, JM, RR two weeks prior to presentation (dept. doesn't need report)
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*Overview Users
**keeping track of users' status in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**5 pending, 3 unknown, 3 completed (+1 pilot), 3 aborted
***pending: AP journalist w/ Paul Ryan correspondence (09/04), molecular/cellular bio researcher with paper abstracts (08/29), York U communications researcher on ~WikiLeaks (08/27)
2. [[An Insight-Based Evaluation of Overview]] with data journalism students - make mention of this in RPE paper?
*study more likely to occur in Winter 2013 with Overview 0.1 Alpha
''New projects'' and collaboration and/or internship placement opportunities
*Ideally avoiding the gatekeeper / limited access to users scenario (϶erview)
3. Tableau internship potential 2013 - upcoming phone/Skype call w/ J. Mackinlay (Sept)
*to schedule
*Mixed-method evaluation of visualization tools/techniques for supporting exploratory data analysisꍺ
>//"I don't think Tableau is very interested in an investment in formal evaluation methodologies because we are having tremendous success with informal evaluation. We are one of those lucky companies that enthusiastically use the software we create. Nevertheless, I would be delighted to talk to Matt."//
4.Task Taxonomy project w/ MS, MM
5. Extending/validating <<cite Buja2009 bibliography:Bibliography>> and <<cite Wickham2010>>'s graphical inference methods
*recent relevant reading:
**<<cite Majumder2010>> - Iowa State Tech Report on graphical inference, used for regression model parameters; [[MTurk|http://mturk.com/]] study
**<<cite Zhao2012>> - Iowa State Tech Report on graphical inference (eye-tracking study)
**<<cite Kairam2012>>: [[notes|Information Visualization: Techniques]] - ~GraphPrism AVI paper, use of graphical inference, also used in <<cite Wood2011>>'s ~BallotMaps paper
*extending method to ~SPLOMs / <<cite Sedlmair2012>>, DR data, follow-on from DRITW: implications for more than 20 plots, p values
*teaching myself R, ggplot (<<cite Wickham2010b>>, qplot, nullabor (<<cite Wickham2010>>)
Upcoming:
*reading SN's CHI draft (09/05)
*~VisWeek 2012 registration (done) (Oct 14 - 19) - BELIV workshop Oct 14/15 (to discuss accommodation / travel w/ MS)
*ACM student membership renewal (done)
*DRITW next steps meeting (date TBD, week of Sept 15)
*~InfoVis group website (Sept 10)
*orientation committee work (first week of Sept.) (mostly done)
*CPSC 508 (operating systems) TR 11:00-12:30
⬥au Pitch:
>My research is about furthering methodologies for evaluating visualization tools and techniques. This involves executing, comparing, and writing about quantitative and qualitative methods used for determining whether a visualization tool or technique is usable, useful, and appropriate for its deployment context. Another aspect of this work is determining at which points these methods are appropriate and informative during the various phases of tool or technique development. Finally, this work will allow me to explore the potential of emerging evaluation methods, such as those presented at recent "Beyond Time and Error" novel evaluation (BELIV) workshops.
>
>Of particular interest to me is how we can design evaluation methodologies that assess how visualization tools support exploratory data analysis and serendipitous discovery. My most recent project is an example of this: a mixed-method evaluation of Overview, a tool for visualizing and exploring large text document sets, built by our collaborators at the Associated Press. My ongoing objective will be to compare the efficacy of evaluation methods used in the Overview project with those used in future projects.
>
>I came to be interested in this area or research during my undergraduate studies in cognitive science, where I was fascinated by graphical perception, working memory, and attention, along with their implications for interaction design. This interest persisted throughout my Master's work in HCI, where I focused on issues relating to task-switching and interruptions. It was during this time that I began to realise the limitations of quantitative methodologies for answering many research questions, and I acknowledged a need for the triangulation of research methods, both quantitative and qualitative, those deployed in controlled settings as well as those deployed "in the wild".
!!References
<<bibliography>>
*Meetings in Sep: TM: 09/05, ''TM: 09/12'', JM: 09/19, TM/TBD: 09/26
1. [[Overview]] Project
*RPE paper draft v4
**supplemental materials: annotated bibliography, EPSE 595 proposal, question list, presentation for JS (includes initial GT code list)
**what did I do?: active voice / more method execution details
***subtler distinction b/w exhaustive search vs. exploratory search
***reflective para in Discussion re: findings in light of larger ~PhD goals
*RPE presentation scheduling:
**tentative Fri Oct 12 - to sort out
**presentation details in grad handbook
**TM/JM to arrange a date/time with RR and recruit a chair from the GAC (dept. invited to presentation)
**MB to send out report to TM, JM, RR two weeks prior to presentation (dept. doesn't need report)
1a. [[Document mining in data journalism]] - post-deployment field data collection from users
*Overview Users
**keeping track of users' status in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**5 pending, 3 unknown, 3 completed (+1 pilot), 3 aborted
***pending: AP journalist w/ Paul Ryan correspondence (09/04), molecular/cellular bio researcher with paper abstracts (08/29), York U communications researcher on ~WikiLeaks (08/27)
***Jenkins (~WikiLeaks) sent request for config help to me last night (cc JS, radio silence on his end)
1b. [[An Insight-Based Evaluation of Overview]] with data journalism students
*study more likely to occur in Winter 2013 with Overview 0.1 Alpha
2.Task Taxonomy project
*New lit review page: [[Task Characterization: Meta Analysis]]
*reading Munzner, Meyer, Sedlmair (2012) BELIV submission, <<cite Heer2012a bibliography:Bibliography>>
*to read: Yi 2007, Zhou/Feiner 1998, Casner 1991, Wehrend/Lewis 1990
*to re-read: Amar/Stasko 2005, Shneiderman 1996
3. Tableau internship potential 2013 - upcoming phone/Skype call w/ J. Mackinlay (Sept)
>//"I don't think Tableau is very interested in an investment in formal evaluation methodologies because we are having tremendous success with informal evaluation. We are one of those lucky companies that enthusiastically use the software we create. Nevertheless, I would be delighted to talk to Matt."//
*emailed last Thursday (no response so far)
4. Extending/validating <<cite Buja2009 bibliography:Bibliography>> and <<cite Wickham2010>>'s graphical inference methods
5. Upcoming:
*reading MH's CHI draft (09/12)
*~VisWeek 2012 registration (done) (Oct 14 - 19) - BELIV workshop Oct 14/15 (accom, travel - done)
*DRITW next steps meeting (date TBD, week of Sept 18)
*CPSC 508 (operating systems) TR 11:00-12:30
!!References
<<bibliography>>
*Meetings in Oct: TM+RR: 10/03, ''TM: 10/10'', (~VisWeek), TM/TBD: 10/24, TBD: 10/31
1. Overview / [[A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]] + RR feedback, MB response, minutes from 10/03 meeting, to-read list updated
*to do: write 1p pitch to UBC journalism / humanities - paired analysis sessions with grad students in journalism/humanities
*cite paired analysis, contextual inquiry as options
*[[Overview users|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]]
*ongoing communication w/ G. Carenini and his ~PhD student
2. [[Task Characterization: Meta Analysis]]
*recently mining the ~GeoVis literature
*added task / user goals taxonomies in other areas of HCI / HII / IR / library / info sciences
*RR's philosophy of scientific discovery literature (see [[to-read|A Preliminary Post-Deployment Evaluation of a Visual Document Mining Tool]])
*recently read ([[notes|Task Characterization: Meta Analysis]]):
**<<cite Rensink2012 bibliography:Bibliography>>
**<<cite Dou2009>>
**<<cite Shrinivasan2008>>
**<<cite Tukey1980>>
**<<cite Roth2012>>
**<<cite Plaisant1995>>
**<<cite Chuah1996>>
**<<cite Tory2004>>
**<<cite Mullins1993>>
**<<cite Ware2004>> (re-read from 533C)
**<<cite Gotz2008>>
**<<cite Casner1991>>
**<<cite Jankun-Kelly2007>>
**<<cite Card1999>> (re-read from 533C)
**<<cite Lam2008a>>
**<<cite Lee2006>>
**<<cite Morse2000>>
**<<cite Roth1990>>
**<<cite Wehrend1990>>
**<<cite Chi1998>>
**<<cite Amar2005>> (re-read from 533C)
**<<cite Kandel2012>> (~InfoVis 09/17)
**<<cite Zhou1998>>
**<<cite Yi2007>>
*TM: secret ways of finding stuff: contact Eugene Barsky, Aleteia Greenwood
*not just a survey paper - proposing a mid-level taxonomy of our own
*~EuroVis Dec. 5 deadline (7 weeks - 2 for ~VisWeek, DRITW, CPSC 508) - ~VisWeek more realistic
*Sorting: what's the level (high/mid/low), method / origin (expert observation / non-expert observation / introspection-reflection), venue, type (objective/operator/operand: <<cite Roth2012>>)
*minor contribution: tasks from IR/HII/LIBR literature
3. Tableau internship potential 2013 - upcoming phone/Skype call w/ J. Mackinlay (Sept) (no response)
4. Upcoming/Recent:
*MAGIC open house tomorrow 4:30-6 in iccs atrium
*~VisWeek, [[BELIV|http://www.beliv.org/wiki/BELIV2012]] workshop Oct 14-19
**to read: [[BELIV 12 position/research papers|References: BELIV]] (sent to TM)
*DRITW next steps meeting (dedicating time to revisions after ~VisWeek)
*CPSC 508 (operating systems) TR 11:00-12:30
*ongoing chatting w/ friend @ Ontario Inst. Cancer Research re: SNP visualization, data cleaning
*attended SLAIS 10/04 lecture: [[Planned Obsolescence: Publishing, Technology, and the Future of the Academy|http://t.co/3e2Z0sH0]] - [[Kathleen Fitzpatrick|http://www.plannedobsolescence.net/]] - [[notes|Journal 12.10.01 - 12.10.05]]
!!References
<<bibliography>>
*Meetings in Nov: TM+JM 11/07, ''TM 11/14'', TM 11/21, TM 11/28, if necessary
1. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*[[Mid level task abstractions for visualization design and evaluation|http://goo.gl/b33pE]] - pre-pre-paper slide deck
**cross-cutting dimensions: abstract -> domain specific, abstract -> interface specific, semantic / syntactic
*Starting to converge, thinking about methodology
**toward midlevel: Springmeyer '92 and Heer/Shneiderman '12 - as starting points, then low-level Amar '05
**not a methodology-first paper, not a GT paper, an introspection paper framed by prior work
*BELIV discussion: arguments for/against a mid-level task taxonomy
*Added some ~VisWeek papers to the to read queue:
**<<cite Crouser2012 bibliography:VisWeek2012>> - on affordances in Vis/VA
**<<cite Pohl2012a>> - comparing high-level theories describing VA process
**<<cite Cottam2012a>> - on describing dynamic vis
**<<cite Mcnamara2012>> - BELIV paper on //Mental Model Formation through Information Foraging//
!References
<<bibliography VisWeek2012>>
*Meetings in Nov: TM+JM 11/07, TM 11/14, ''TM 11/21'', TM 11/28
1. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*[[Mid level task abstractions for visualization design and evaluation|http://goo.gl/b33pE]] - pre-pre-paper slide deck (12.11.14)
**cross-cutting dimensions: abstract -> domain specific, abstract -> interface specific, semantic / syntactic
**toward midlevel: <<cite Springmeyer1992 bibliography:Bibliography-TaskTaxonomy>> and <<cite Heer2012a>> as starting points,
***then low-level <<cite Amar2005>>, <<cite Gotz2008>>, <<cite Chuah1996>>
**not a methodology-first paper, not a GT paper, an introspection paper framed by prior work
*Added some ~VisWeek papers to the to read queue:
**<<cite Crouser2012>> - on affordances in Vis/VA
**<<cite Pohl2012a>> - comparing high-level theories describing VA process
**<<cite Cottam2012a>> - on describing dynamic vis
**<<cite Mcnamara2012>> - BELIV paper on //Mental Model Formation through Information Foraging//
!References
<<bibliography VisWeek2012>>
*Meetings in Nov: TM+JM 11/07, TM 11/14, TM 11/21, ''TM 11/28''
[[Task Taxonomy|Task Characterization: Meta Analysis]]
*[[Mid level task abstractions for visualization design and evaluation|http://goo.gl/b33pE]] - pre-pre-paper slide deck (12.11.14)
**cross-cutting dimensions: abstract -> domain specific, abstract -> interface specific, semantic / syntactic
**12.11.23 version in SVN
It turns out that there's little agreement at level of high-level models too�books cited by <<cite Pohl2012a bibliography:Bibliography>> (Proc. VAST '12):
*Carroll, J.M. (ed.) (2003). [[HCI Models, Theories and Frameworks|http://www.amazon.com/HCI-Models-Theories-Frameworks-Multidisciplinary/dp/1558608087/ref=sr_1_1?ie=UTF8&qid=1354062239&sr=8-1&keywords=HCI+Models%2C+Theories+and+Frameworks]]. Morgan Kaufmann
**Perry, M. //Distributed cognition//
**Pirolli. P. //Exploring and finding information// (sensemaking)
*Davidson, J. E. and Sternberg, J. R. (eds.) (2003). [[The Psychology of Problem Solving|http://www.amazon.com/Psychology-Problem-Solving-Janet-Davidson/dp/0521797411/ref=sr_1_1?s=books&ie=UTF8&qid=1354062265&sr=1-1&keywords=The+Psychology+of+Problem+Solving]]. Cambridge University Press
**Davidson, J. E. //Insights about insightful problem solving// (gestalt theory)
**Pretz, J. E. and Naples, J. //Recognizing, defining, and representing problems// (gestalt theory)
*Freedle, R. (ed.) (1990). [[Artificial Intelligence and the Future of Testing|http://www.amazon.com/Artificial-Intelligence-Future-Testing-Freedle/dp/0805801170/ref=sr_1_1?s=books&ie=UTF8&qid=1354062285&sr=1-1&keywords=Artificial+Intelligence+and+the+Future+of+Testing]]. Lawrence Erlbaum
**Pinker, S. //A theory of graph comprehension//
*Holyoak, K. J. and Morrison, R. G. (eds.) (2005, 2007). [[The Cambridge Handbook of Thinking and Reasoning|http://www.amazon.com/Cambridge-Handbook-Reasoning-Handbooks-Psychology/dp/0521531012/ref=sr_1_1?s=books&ie=UTF8&qid=1354062310&sr=1-1&keywords=The+Cambridge+Handbook+of+Thinking+and+Reasoning]]. Cambridge University Press
**Novick, L. R. and Bassok, M. //Problem solving// (gestalt theory)
**Tversky, B. //Visuospatial reasoning// (graph comprehension)
*Hutchins, E. (1995). [[Cognition In The Wild|http://www.amazon.com/Cognition-Bradford-Books-Edwin-Hutchins/dp/0262581469/ref=sr_1_1?s=books&ie=UTF8&qid=1354062328&sr=1-1&keywords=Cognition+In+The+Wild]]. The MIT Press
*Mirel, B. (2004). [[Interaction Design for Complex Problem Solving|http://www.amazon.com/Interaction-Design-Complex-Problem-Solving/dp/1558608311/ref=sr_1_1?s=books&ie=UTF8&qid=1354062352&sr=1-1&keywords=Interaction+Design+for+Complex+Problem+Solving]]. Elsevier, Morgan Kaufmann
*Shah, P. and Miyake, A. (eds.) (2005). [[The Cambridge Handbook of Visuospatial Thinking|http://www.amazon.com/Cambridge-Handbook-Visuospatial-Handbooks-Psychology/dp/0521807107/ref=sr_1_1?s=books&ie=UTF8&qid=1354062611&sr=1-1&keywords=The+Cambridge+Handbook+of+Visuospatial+Thinking]]. Cambridge University Press
*Reason, J. (1990). [[Human Error|http://www.amazon.com/Human-Error-James-Reason/dp/0521314194/ref=sr_1_1?ie=UTF8&qid=1354065753&sr=8-1&keywords=reason+human+error]]. Cambridge University Press
*Rogers, Y. and Rutherford, A. and Bibby, P.A. (eds.) (1992). [[Models in the Mind: Theory Perspective and Application|http://www.amazon.com/Models-Mind-Perspective-Application-Computers/dp/0125929706/ref=sr_1_1?s=books&ie=UTF8&qid=1354062368&sr=1-1&keywords=Models+in+the+Mind%3A+Theory+Perspective+and+Application]]. Academic Press
**O'Malley C. and Draper S. //Representation and interaction: Are mental models all in the mind// (distributed cognition)
*Salomon, G. (ed.) (1993). [[Distributed Cognition: Psychological and Educational Considerations|http://www.amazon.com/Distributed-Cognitions-Psychological-Considerations-Computational/dp/0521574234/ref=sr_1_1?s=books&ie=UTF8&qid=1354062389&sr=1-1&keywords=Distributed+Cognition%3A+Psychological+and+Educational+Considerations]]. Cambridge University Press
*Sternberg, R. J. and Davidson, J. E. (eds.) (1995). [[The Nature of Insight|http://www.amazon.com/Nature-Insight-Bradford-Books/dp/0262691876/ref=sr_1_1?s=books&ie=UTF8&qid=1354062415&sr=1-1&keywords=The+Nature+of+Insight]]. The MIT Press
**Dominowski, R. L., and Dallob, P. //Insights and problem solving// (gestalt theory)
**Weisberg, R.W. //Prolegomena to theories of insight in problem solving: A taxonomy of problems// (gestalt theory)
**Mayer, M. //The search for insight: Grappling with gestalt psychology's unanswered questions//
!References
<<bibliography Bibliography>>
*Meetings in Dec: JM 12/12, ''TM 12/19'' 2 - (3:30 pm PT, 5 - 6:30 pm ET via Skype)
1. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*Mid level task abstractions for visualization design and evaluation - pre-pre-paper slide deck (12.12.19) (in SVN)
*Reading:
**<<cite Pohl2012a bibliography:Bibliography>> (Proc. VAST) comparing higher-level theories and frameworks for VA: [[notes|Task Characterization: Meta Analysis]]: sensemaking theories, gestalt theories (insight), distributed cognition theories, graph comprehension theories, skill-rule knowledge theories
**<<cite Liu2008>> (TVCG) on distributed cognition as framework for VA: [[notes|Task Characterization: Meta Analysis]]
**<<cite Hollan2000>> (TOCHI) on distributed cognition in HCI: ''pragmatic'' and ''epistemic'' actions [[notes|Task Characterization: Meta Analysis]]
**<<cite Kirsh2006>> on a distributed cognition methodology: [[notes|Task Characterization: Meta Analysis]]
**<<cite Crouser2012>> (Proc. VAST) on affordance-based framework for VA: [[notes|Task Characterization: Meta Analysis]]
*Further reading, some books cited by <<cite Pohl2012a>>:
**Hutchins, E. (1995). [[Cognition In The Wild|http://www.amazon.com/Cognition-Bradford-Books-Edwin-Hutchins/dp/0262581469/ref=sr_1_1?s=books&ie=UTF8&qid=1354062328&sr=1-1&keywords=Cognition+In+The+Wild]]. The MIT Press
**Mirel, B. (2004). [[Interaction Design for Complex Problem Solving|http://www.amazon.com/Interaction-Design-Complex-Problem-Solving/dp/1558608311/ref=sr_1_1?s=books&ie=UTF8&qid=1354062352&sr=1-1&keywords=Interaction+Design+for+Complex+Problem+Solving]]. Elsevier, Morgan Kaufmann
**Sternberg, R. J. and Davidson, J. E. (eds.) (1995). [[The Nature of Insight|http://www.amazon.com/Nature-Insight-Bradford-Books/dp/0262691876/ref=sr_1_1?s=books&ie=UTF8&qid=1354062415&sr=1-1&keywords=The+Nature+of+Insight]]. The MIT Press
***//Insights and problem solving//
***//Prolegomena to theories of insight in problem solving: A taxonomy of problems//
***//The search for insight: Grappling with gestalt psychology's unanswered questions//
2. Overview
*Interviewed Overview (desktop version) user 12.05: AP's J. Gillum on ~FOIAs re: Paul Ryan's correspondence (~8k pages), usage differed from previous Overview user
**data collection limited (incomplete log files, interview recording SNAFU), but enough notes, data + tag files, and email correspondence to go on
*to do: Overview web version log file efficacy review, consider JS' 11.21 email on Overview development, why the web version is attracting no users
3. Misc.
*DRITW draft editing & submission; action items from 12.12.19 ~InfoVis discussion
*CPSC 508 project - submitting today/tomorrow
*TOCHI submission w/ CT, JM rejected :(
*Selected as CHI '13 SV - Apr. 26 - May 2
*Reading:
**<<cite Kuhn1962>> on scientific revolutions
!References
<<bibliography Bibliography>>
*Meetings in August: ''08/08'', 08/15, 08/22, 08/29
1. Pulse
*Mitacs application submitted (awaiting response)
*Granted access to in-dev demo version of [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]] (13.07.22)
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity in terms of //how// and //what//, but not //why// (task analysis will put all of this together); *partially done*
*Pulse Energy Manager user interviews:
**met w/ [[CG, SES energy consultant|Pulse-SES-CG-13.08.06]] 13.08.06: [[notes|Pulse-SES-CG-13.08.06]], CG to send Excel energy analysis docs
**met w/ [[JC, McGill EM|Pulse-JC-13.07.29-MEB-13.08.03]] 13.07.29 and [[MEB, McGill Energy Project developer|Pulse-JC-13.07.29-MEB-13.08.03]] 13.08.03: [[notes|Pulse-JC-13.07.29-MEB-13.08.03]]
**met w/ [[LZ, UBC ECM|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]] 13.07.09: [[meeting + documentation notes|Pulse-LZ-13.07.09]]
*future energy manager meetings
**KT sent UC Berkeley energy manager contact - available in mid-August for interview
**awaiting response from UBC BSM Ops. manager (out-of-office vacation message)
**KT to introduce energy managers at ~UVic, non-university energy managers
2. Task Typology
*final camera-ready edit request 08/07 re: ࡣing on p1, author bullets, email comma separation (done)
*Schulz et al.'s //Design Space of Vis. Tasks// paper, implications for presentation
*~InfoVis group practice talk Oct 9; (MUX practice talk?)
3. Overview
*timeline: pre-paper talk for ~InfoVis group Aug 14, draft for JS between Aug 21 and Sept 4, draft for SI (+JM?) Sept 4, draft for ~InfoVis (+MUX?) Sept 11, draft submitted Sept 18
*[[notes from yesterday's meeting w/ JS|TM-JS-SI-MB-13.08.07]]
**watching JS' [[NewsU|https://twitter.com/newsu]] webinar [[Document Mining with Overview: A Digital Tools Tutorial|http://www.newsu.org/digital-tools-overview]]
*implications of latest ~MoDisco rejection for CHI paper
*implications of DRITW limbo for CHI paper, esp. re: "why do people like to look at dots of DR data?"
*case studies:
**to interview Sue, Dallas journalist, Aug 19-21
**partial transcript / notes document for the MK interview: [[Overview - MK interview transcript/notes]] (Feb 2013)
**other interview notes/transcripts: [[Overview - JG interview notes]] (Dec 2012), [[Overview - JW interview transcript]] (June 2012)
4. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions, submission date Oct 4, planning to have draft Oct 1-2
**current working title: //~Multi-Level Abstract Task Analysis for Visualization Design and Evaluation//
**4p extended abstract + 1䡴ement of expected benefit, **letter of nomination from TM**, current CV
5. Misc.
*learning d3 re: [[scatterplot focus and context|http://jsfiddle.net/PyvZ7/7/]]
*attended [[presentation by visualization designer/artist Jer Thorp|http://about.me/jerthorp]] @ BCCDC 07/26
*checked out [[Bret Victor|http://worrydream.com/#]]'s website (via [[@EdwardTufte|http://twitter.com/EdwardTufte]])
*watched [[@jonathanstray|http://twitter.com/jonathanstray]] on [[CSPAN|http://www.c-span.org/Live-Video/C-SPAN/]] re: NSA surveillance 06/08
*watched [[satisfy the cat|http://www.youtube.com/watch?v=dln9xDsmCoY]]
*read [[nanocubes|http://www.nanocubes.net/]], [[imMens|http://vis.stanford.edu/papers/immens]], and [[bigvis|http://vita.had.co.nz/papers/bigvis.pdf]] for ~InfoVis group
*TOCHI paper w/ CT, JM: token to CT 08/01/13
*In Ontario Sept 19-Oct 1, (2/3 vacation, 1/6 CHI DC, 1/6 Task Typology talk prep)
*Meetings in August: 08/08, 08/16, ''08/29''
1. Pulse
*Mitacs application approved with minor revisions
**Revisions due in 90 days (Nov 24)
**Suggested revisions:
>//"It is not clear whether household consumers are consulted to obtain feedbacks."// (㩊>
>//"If helping consumers to reduce invoice cost is an objective, feedbacks from the consumer group should be included in the later evaluation process as well."// (䩊>
>//"On page 9 in the last line, it says that 鮡lly, ⩴e�olves identifying and writing about the research contributions ...�general, research contributions are defined at the outset as a part of the motivation. It should not be left to the end of the project. Either this task is moved to an early stage or explanation is required on why 峥arch contributions� left to the end."// (婊>
>//"32 design study pitfalls are mentioned in the next sentence. In order to demonstrate the applicants뮯wledge of resolving these issues, I suggest that a few major pitfalls are included and solutions proposed in this application."// (婊>
>//"Similarly, in the subsequent paragraph, I suggest the applicants be more specific when saying வmber of design and evaluation methods will be adopted at the four core stages of the methodology.�婊*Pulse Energy Manager user interviews:
**to meet w/ BE, UBC Building Operations, 13.09.05
**met w/ KN, UC Berkeley energy analyst, 13.08.28 (+ video / to do: transcript/notes)
**met w/ [[CG, SES energy consultant|Pulse-SES-CG-13.08.06]] 13.08.06: [[notes|Pulse-SES-CG-13.08.06]], CG to sent Excel energy analysis docs last week (to do: wrangle giant Excel sheets into viewable format)
**met w/ [[JC, McGill EM|Pulse-JC-13.07.29-MEB-13.08.03]] 13.07.29 and [[MEB, McGill Energy Project developer|Pulse-JC-13.07.29-MEB-13.08.03]] 13.08.03: [[notes|Pulse-JC-13.07.29-MEB-13.08.03]]
**met w/ [[LZ, UBC ECM|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]] 13.07.09: [[meeting + documentation notes|Pulse-LZ-13.07.09]]
**KT to introduce energy managers at ~UVic, non-university energy managers
*Granted access to in-dev demo version of [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]] (13.07.22)
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity; *partially done*
2. Task Typology
*VIS [[Schedule-at-a-glance|http://ieeevis.org/attachments/2013_VIS-at-a-glance.pdf]]: likely session "defining the design space" Tuesday 4:15pm
*VIS [[fast forward slides/video due Sept 24|http://www.cs.toronto.edu/~fchevali/vis2013ffw/pmwiki.php?n=Main.HomePage]]
*Schulz et al.'s //Design Space of Vis. Tasks// paper, implications for presentation (given title of session, Schulz likely to go first?)
*~InfoVis group practice talk Oct 9
3. Overview
*paper: high-level feedback and thoughts on CHI vs. ~EuroVis
*timeline (if CHI): partial draft today, full draft next week (token to JS, SI?), ~InfoVis group Sept 11, MUX Sept 11
*case studies:
**interviewed SA, Dallas journalist, Aug 22 (off-the-record, WIP)
**notes: [[Overview - MK interview transcript/notes]] (Feb 2013), [[Overview - JG interview notes]] (Dec 2012), [[Overview - JW interview transcript]] (June 2012)
4. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions, submission date Oct 4, planning to have draft Oct 1-2
**current working title: //Mapping Domain Problems to Techniques: Task Abstractions in Visualization Design and Evaluation//
**4p extended abstract + 1p statement of expected benefit, **letter of nomination from TM** (reminder to be sent Sept. 18), current CV
5. Misc.
*In Ontario Sept 19-Oct 1 (Vacation Sept 23-27)
*Meetings in September: ''09/05'', 09/12 (w/ L. Wilkinson), 09/18 1-3pm
1. Pulse
*Mitacs application approved with minor revisions
**TM feedback on 09/03 revisions
**Revisions due in 90 days (Nov 24)
*[[Creative User-Centered Visualization Design for Energy Analysts and Modelers|http://openaccess.city.ac.uk/2618/1/CreativeUserCentredVisDesign.pdf]] by Goodwin, Dykes, Jones, Dillingham, Dove, Duffy, Kachkaev, Slingsby, & Wood (2013). To appear at IEEE ~InfoVis 2013 (author pre-print)
*Met w/ JP 09/04 to discuss funding / tuition / NSERC + 4YF
*Pulse Energy Manager user interviews:
**to meet w/ NV, SES, 13.09.16
**met w/ BE, UBC Building Operations, 13.09.05 (to do: digitize notes)
**met w/ KN, UC Berkeley energy analyst, 13.08.28 (+ video / to do: transcript/notes)
**met w/ [[CG, SES energy consultant|Pulse-SES-CG-13.08.06]] 13.08.06: [[notes|Pulse-SES-CG-13.08.06]], CG to sent Excel energy analysis docs last week (to do: wrangle giant Excel sheets into viewable format)
**met w/ [[JC, McGill EM|Pulse-JC-13.07.29-MEB-13.08.03]] 13.07.29 and [[MEB, McGill Energy Project developer|Pulse-JC-13.07.29-MEB-13.08.03]] 13.08.03: [[notes|Pulse-JC-13.07.29-MEB-13.08.03]]
**met w/ [[LZ, UBC ECM|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]] 13.07.09: [[meeting + documentation notes|Pulse-LZ-13.07.09]]
**KT to introduce energy managers at Capilano, ~UVic, non-university energy managers (pinged 09/03): reply 09/05: no-go on utility manager project (no access to users, data)
*Granted access to in-dev demo version of [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]] (13.07.22)
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity; *partially done*
2. Overview
*currently re-writing draft, focusing on Analysis and Discussion sections
*converted to ~EuroVis format (<10p)
*timeline: token to TM Sun 09/08, to MB Wed 09/11, JS on Thu 09/12, to ~InfoVis group Mon 09/16-17, ~InfoVis group read 09/18: to do: give JS heads up
*case studies:
**interviewed SA, Dallas journalist, Aug 22 (off-the-record, WIP, to follow-up when completed
**notes: [[Overview - MK interview transcript/notes]] (Feb 2013), [[Overview - JG interview notes]] (Dec 2012), [[Overview - JW interview transcript]] (June 2012)
3. Task Typology
*Registered 09/05 ($360), renewed ACM student membership ($60)
*Budget: airfare, accommodation (conference hotel sold out, to split w/ JF?), per diem
*VIS [[Schedule-at-a-glance|http://ieeevis.org/attachments/2013_VIS-at-a-glance.pdf]]: likely session "defining the design space" Tuesday 4:15pm
*VIS video preview: [[draft|http://goo.gl/b1uqob]]
*VIS [[fast forward slides/video due Sept 24|http://www.cs.toronto.edu/~fchevali/vis2013ffw/pmwiki.php?n=Main.HomePage]] - likely a similar script to video preview, to give practice FF to ~InfoVis group Sep 18: why do you care?
*~InfoVis group practice talk Oct 9 (to do: prepare and rehearse talk)
4. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions, submission date Oct 4, planning to have draft Oct 1-2
**current working title: //Mapping Domain Problems to Techniques: Task Abstractions in Visualization Design and Evaluation//
**4p extended abstract + 1p statement of expected benefit, **letter of nomination from TM** (reminder to be sent Sept. 18), current CV
5. Misc.
*~InfoVis group pubs page
*to meet w/ L. Wilkinson 09/12 4pm
*to meet w/ S. North 09/16 10am
*In Ontario Sept 20-30 (Vacation Sept 23-27)
*Meetings in Jan: '''TM 01/09'' 11am, JM 01/16 (early pre-paper talk draft), TM 01/23 (pre-paper talk prep), TM 01/30 (pre-paper talk debrief)
1. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*Mid level task abstractions for visualization design and evaluation - pre-pre-paper slide deck (13.01.09) (in SVN)
*pre-paper talk 01/28
*Reading:
**<<cite Case2008 bibliography:Bibliography>> on [[models of information seeking|Task Characterization: Meta Analysis]] - ch. 6 (JD's LIBR 553 ref)
**<<cite Case2008>> on [[theories, perspectives, paradigms|Task Characterization: Meta Analysis]] - ch. 7
**<<cite Toms1999>> on serendipitous information retrieval
**<<cite Toms2000>> on understanding and facilitating browsing: [[notes|Task Characterization: Meta Analysis]] (journal version of <<cite Toms1999>>)
**<<cite Hutchins1995>> on Distributed cognition (//Cognition in the Wild//, ch. 1: welcome aboard)
2. Overview
*Interviewed Overview (desktop version) user 12.05: AP's J. Gillum on ~FOIAs re: Paul Ryan's correspondence (~8k pages), usage differed from previous Overview user
**data collection limited (incomplete log files, interview recording SNAFU), but enough notes, data + tag files, and email correspondence to go on
*back in touch w/ Overview desktop user in the UK, using it to mine tweets
*JS on logging (e.g. [[my vancouver news dataset's log|https://www.overviewproject.org/documentsets/305/log-entries]]), new user story w/ Overview: [[The Best of Our Gun Debate: Readers Weigh In on Owning Firearms|http://www.thedailybeast.com/articles/2012/12/22/the-best-of-our-gun-debate-readers-weigh-in-on-owning-firearms.html]] and [[The State-by-State Breakdown|http://www.thedailybeast.com/articles/2012/12/21/the-best-of-our-gun-debate-the-state-by-state-breakdown.html]] (12.21.12)
**JS experimenting with some different clustering algorithms
**re: user study (w/ students):
>''JS:'' //Re the user study: I think we're getting close the point where that's reasonable. We're struggling with infrastructural issues at the moment which I'd like to see resolved.//
>
>礠like to wait until post-NICAR to do a study so we can go beyond 10k docs. After that, it would depend on whether we can use ~DocumentCloud, bearing in mind that DC becomes less viable as the docset size increases. I'd like to make it through that by mid summer.//
>
>//Institutionally, I am teaching Jan 14-Feb 10 at HKU, and again in the fall at Columbia. What would your timeframe be for such a study?//
3. Misc.
*TOCHI submission w/ CT, JM rejected :(
*Selected as CHI '13 SV - Apr. 26 - May 2
**[[CHI '13 workshops|http://chi2013.acm.org/authors/call-for-participation/workshop-participants/]]
***[[evaluation methods for creativity support environments|http://ecologylab.net/workshops/creativity/]]
***[[Many People, Many Eyes: Aggregating Influences of Visual Perception on User Interface Design|http://people.seas.harvard.edu/~reinecke/manyeyes/]]
*meeting w/ R. Leung (SAP) 01/19 re: VA research
*checking out Coursera:
**[[Stanford ML|https://www.coursera.org/course/ml]]
**[[UW Data Science|https://www.coursera.org/course/datasci]]
*Joined [[HxD Vancouver|http://www.meetup.com/HXD-Vancouver/t/wm1?rv=wm1&ec=wm1]] meetup group, [[Vancouver Data Visualization|http://www.meetup.com/Vancouver-Data-Visualization/t/wm1?rv=wm1&ec=wm1]] group
*[[IBM research opportunities|http://ibm-research.jobs/cambridge-ma/research-summer-intern-cambridge/33144214/job/]]
*Adobe research opportunities
*Other Reading:
**<<cite Kuhn1962>> on scientific revolutions
**<<cite Micallef2012>> - Proc. ~InfoVis '12 on Bayesian reasoning for ~InfoVis group discussion: [[notes|Information Visualization Evaluation: Quantitative Methods]]
**[[2012: The Year in Graphics|http://www.nytimes.com/interactive/2012/12/30/multimedia/2012-the-year-in-graphics.html]] - NYT
**[[Census Dotmap|http://bmander.com/dotmap/index.html]] by Brandon ~Martin-Anderson
!References
<<bibliography Bibliography>>
*Meetings in Jan: TM 01/09 11am, JM+TM 01/16 (early pre-paper talk draft), TM 01/17 (pre-paper talk prep), TM 01/21 (pre-paper talk debrief), ''TM 01/30''
1. Potential research internships / collaborations summer '13:
*[[IBM research opportunities|http://ibm-research.jobs/cambridge-ma/research-summer-intern-cambridge/33144214/job/]] - to do: MB to apply,TM to introduce to AP
*[[MSR internships|https://research.microsoft.com/apps/tools/jobs/intern.aspx]] w/ [[CUE|http://research.microsoft.com/en-us/um/redmond/groups/cue/#people]], [[VIBE|http://research.microsoft.com/en-us/um/redmond/groups/vibe/vibewebpage/#listName=peopleList&id=80389]], [[CLUES|http://research.microsoft.com/en-us/groups/clues/]] - application submitted, to do: MS, RR, TM to submit references (MB to write prelim draft for RR), TM to read draft letters to SD, MC, TM to introduce to BL, DF
*[[Google UX Research internship|http://www.google.com/intl/en/jobs/students/tech/internships/uscanada/user-experience-research-intern-summer-2013-north-america.html]] - HL 01.28 response
>''HL'': // I am not aware of any work on evaluation or task characterization in visualization. In most cases, visualization internships are very implementation-centric, so if you want to do more user evaluation work, UX may be a better choice.//
>
>//I am currently in the structured data team that aims to make web tables accessible to Google users (visualization increases the accessibility). I will also be collaborating with Ed Chi on visualizing social data.//
*meeting w/ R. Leung 01/19 re: VA research @ SAP
*meeting w/ MS + TM on follow-up work to DRITW, cluster separation factors studies (not yet scheduled)
2. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*01.30.13 keynote slides/PDF in SVN
*Reading recently:
**<<cite Fisher2012b bibliography:Bibliography>> on research directions w.r.t. HCI and big data analytics
**<<cite Kosara2013>> on storytelling and vis
**<<cite Buxton1986>> on chunking and phrasing tasks: [[notes|Task Characterization: Meta Analysis]]
3. TOCHI Review request
*to do - read/review/debrief w/ JM
4. ~C-TOC interruption follow-up study w/ CJ + visiting student
*coding pass done, awaiting pilot results from CJ
5. rejected TOCHI "interruptions in the wild" submission
*New venue? to discuss w/ ST 02.01
6. Overview
*new user story w/ Overview-web: [[The Best of Our Gun Debate: Readers Weigh In on Owning Firearms|http://www.thedailybeast.com/articles/2012/12/22/the-best-of-our-gun-debate-readers-weigh-in-on-owning-firearms.html]] - contacted him re: interview / logs - got a positive response but then silence after suggesting dates/times for interviews
*back in touch w/ Tempero social media data analysis user from the UK - he will be using the web version this month
7. Misc.
*Selected as CHI '13 SV - Apr. 26 - May 2 (travel week before/after - haven't decided yet)
*sitting in on a 8-week [[Data Analysis|https://www.coursera.org/course/dataanalysis]] on Coursera 01.22
*Attended a [[HxD Vancouver|http://www.meetup.com/HXD-Vancouver/t/wm1?rv=wm1&ec=wm1]] meetup 01.24
*attended [[Towards Personal Visual Analytics]] - guest presentation by Sheelagh Carpendale, University of Calgary (01/15/13)
*Out of town over reading week (Feb 18-22, working remotely)
!References
<<bibliography Bibliography>>
*Meetings in April: '''TM 04/03'' 11am, TM 04/10
0. Summer meeting schedule
1. Overview
*interviewed web version user in early February
*usability evaluation / think-aloud studies:
>''JS'' 03.20.13: //"We've more or less reimplemented what we had in the prototype in production. Some big differences of course: different viz and clustering algorithm. But we believe the product is usable. //
>
>//However, no one is using it. We have added analytics last week, in advance of the training webinar, to try to figure out where we are losing people. It looks like most folks never even import an example document set. If we don't have people using the system, post-use interviews are not going to be very helpful.//
>
>//Therefore I'd like to try a different approach: I want to see what naive users do on their first interaction with the system. The methodology I'm imagining is, put them in front of the site and tell them to "try out Overview. If you have a document set, upload it. Otherwise try one of the examples." Ask them to think aloud in the usual way, but say nothing. Do a screen recording.//
>
>//This is really classic usability stuff; hopefully users will get to the visualization and we will learn important things about how users learn the clustering/tagging model -- or even how they hope or expect that things are clustered. I'm lining up a couple journalists here to do this. Do you want to try to do a couple such studies at your end?"//
>''MB'': "//Yes, I can work on arranging some think-aloud studies on my end. We're currently approaching the VisWeek deadline (Mar 31), so once my paper is submitted, I'll be able to dedicate more blocks of time to this.//
>
>//I'll get in touch with the contacts I've made at the Vancouver Sun, Seattle Times, and at the UBC School of Journalism. I'll try polling some of the Vancouver Meetup groups I'm a member of as well. I'll also see if there's any keen students in the Vancouver Institute for Visual Analytics undergrad training program, to whom I've demoed the desktop version of Overview last summer - they often have datasets and time at their disposal. This could be a good opportunity to pilot an eventual think-aloud study with your data journalism students in the 2013-14 academic year."//
>''JS'':"//Thanks Matthew. This is wonderful. Lack of users is the biggest problem we have right now. Question is, why? At worst, we are solving the wrong problem. But I suspect usability. Also, I need feedback on how naive users imagine the task *should* work."//
*ethics amendments needed?
2. Thesis proposal
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*Contents:
**1. Task Taxonomy
**2. Overview (evaluative validation)
**3. Design Study (generative validation)
**4. Other? DRITW? Internship projects?
*Guidance/tips? example proposals from past students?
*Supervisory committee? role of JM?
*Annual progress reports:
**[[CS department|https://www.cs.ubc.ca/students/grad/policies/annual-phd]] (end of April)
**[[NSERC|https://www.grad.ubc.ca/forms/annual-progress-report-fellowship-holders]] (June 1)
3. Internships
*To meet w/ MSR VIBE's DF @ CHI
*Other UX research internships:
**IBM x3 (social media data analysis, healthcare system analysis and simulation)
***to do: email AP re: internship positions in 2013-14, meet at CHI?
**Google (general, not w/ HL)
**Facebook
**Salesforce (position has since been filled)
**Autodesk design research
**Adobe Research
**[[M. Dontcheva|http://www.adobe.com/technology/people/san-francisco/mira-dontcheva.html]] - instructional design, search and sensemaking interfaces, and creativity
**[[B. Suh|http://www.adobe.com/technology/people/san-jose/bongwon-suh.html]] - social media systems, information visualization, sense-making interfaces, and web-scale data mining
4. Misc.
*TOCHI journal paper rewrite, return to data analysis (found an error in how the timing log was implemented)
*Upcoming travel: April 17 - May 3 (CHI '13 SV - Apr. 26 - May 2)
*[[CHI '13 workshops|http://chi2013.acm.org/authors/call-for-participation/workshop-participants/]]
**[[evaluation methods for creativity support environments|http://ecologylab.net/workshops/creativity/]]
**[[Many People, Many Eyes: Aggregating Influences of Visual Perception on User Interface Design|http://people.seas.harvard.edu/~reinecke/manyeyes/]]
*Meetings in April: TM 04/03, TM ''04/10'' 11am
1. Overview
*See 04/08 JS email (TM cc'd) re: [[Contextual Inquiry]] / [[Think-Aloud|Think Aloud Protocol]], ICIJ offshore investment data document dump, Overview study w/ students
*email conversation initiated with C. Phillips, data enterprise editor @ Seattle Times
*emails sent to Vancouver Sun / UBC journalism contact, VIVA; discussion board post to Vancouver [[Hacks/Hackers meetup group|http://www.meetup.com/HacksHackersVancouver/]]
*ICIJ offshore banking document dump - watching [[CBC exposԮ Merchant|http://www.cbc.ca/player/News/TV+Shows/The+National/ID/2369731413/]] by [[F. Zalac|http://www.icij.org/journalists/frederic-zalac]] (CBC, ICIJ - based out of Vancouver)
2. Design studies
*Discussion of possible design study collaborators
3. Thesis proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*Annual progress reports (see drafts):
**[[CS department|https://www.cs.ubc.ca/students/grad/policies/annual-phd]] (end of April)
**[[NSERC|https://www.grad.ubc.ca/forms/annual-progress-report-fellowship-holders]] (June 1)
4. Internships
*To meet w/ MSR VIBE's DF @ CHI
*Other UX research internships:
**IBM x3 (social media data analysis, healthcare system analysis and simulation)
**Google (general, not w/ HL)
**Facebook
**Autodesk design research
**Adobe Research: [[M. Dontcheva|http://www.adobe.com/technology/people/san-francisco/mira-dontcheva.html]], [[B. Suh|http://www.adobe.com/technology/people/san-jose/bongwon-suh.html]]
5. Misc.
*TOCHI journal paper rewrite, return to data analysis (found an error in how the timing log was implemented)
*2x IEEE ~InfoVis '13 paper reviews (due 05/09)
*Upcoming travel: April 17 - May 3 (CHI '13 SV - Apr. 26 - May 2)
*Meetings in May: '''TM 05/014'' 10:30am PDT (12:30pm CDT), committee meeting 05/23 11am x530
1. Pulse Energy
*Visited Pulse's Vancouver office 05/08, met w/ KT and JK
*Potential design study collaboration opportunities TBD (see 05/09 email exchange): see [[Pulse Energy Manager|http://www.pulseenergy.com/pulse-platform/pulse-energy-manager/]]
*potential for [[MITACS Accelerate|http://www.mitacs.ca/accelerate/program-guide]] funding?
**possible meeting w/ TM 05 22-30 (TBD w/ Pulse)
2. Overview
*Re: usability / think-aloud studies:
>//I'll get in touch with the contacts I've made at the Vancouver Sun, Seattle Times, and at the UBC School of Journalism. I'll try polling some of the Vancouver Meetup groups I'm a member of as well. I'll also see if there's any keen students in the Vancouver Institute for Visual Analytics undergrad training program, to whom I've demoed the desktop version of Overview last summer - they often have datasets and time at their disposal. This could be a good opportunity to pilot an eventual think-aloud study with your data journalism students in the 2013-14 academic year."//
*email conversation initiated with C. Phillips, data enterprise editor @ Seattle Times (last heard from Apr. 8, responses sent Apr 9, May 6)
*Current status: no responsesﵲnalism student study timeline TBD (Fall '13/ Winter '14)
*~MoDiscoTag v2 + Overview case study (CHI '14 (Toronto, ~PacificVis '14 (Yokohama)?) TBD w/ SI
**re-read <<cite Gonzalez2003 bibliography:Bibliography>>
3. Internship search (long-term)
*2 week delayed initial response from Hallam, nothing since my return from CHI (reminder sent 05/06)
*Spoke w/ DF (MSR VIBE) at CHI. Deploying an early version (sans-vis) internally this summer, building visualizations. Envisions a larger-scale evaluation of visualizations next summer.
*Spoke w/ GS (formerly UBC Imager, now on the Google UX researcher on Privacy team), introduced to [[J. Staddon|http://home.jessicastaddon.com/]] potential for in visualization internship position next summer
*Bloomberg has a new research internship program beginning this summer (including UX research); will be seeking applicants for next summer in the fall
4. Thesis Proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*Annual progress reports:
**[[CS department|https://www.cs.ubc.ca/students/grad/policies/annual-phd]] (end of April) (meeting 05/23 11am)
***role of JM (supervisor / committee) 05/23, 12-12:30
**[[NSERC|https://www.grad.ubc.ca/forms/annual-progress-report-fellowship-holders]] (June 1)
5. Misc.
*my [[CHI '13 reference roundup|References: CHI2013]]
*appointed to the dept. grad recruitment committee (2013-14)
**clear other czarships for Jan-Mar, particularly IMAGER social czarship (see czar of czars to change czarships by then)
*DRITW revisions (MB token until mid this week)
*wrote 2 IEEE ~InfoVis '13 paper reviews (incl. one on energy usage analysis)
*M.Sc follow-up TOCHI journal paper (w/ CT) rewrite, return to data analysis (found an error in how the timing log was implemented)
*H. Wickham visit
**attended Vancouver R user group Meetup 05/08 @ ~HootSuite: [[Packages R Easy|http://bit.ly/pkgsrez]]
**attended Statistics dept. seminar 05/09: [[Bigvis: visualizing 100,000,000 observations in R]]
**attended Statistics dept. [[ggplot2 workshop|http://bit.ly/ggplot2ubc]] 05/10
*attended M.Sc presentation on user-centred data warehousing and a visual schema exploration tool (RP's student) (05/09)
*reading:
**[[A More Thoughtful but No More Convincing View of Big Data|http://www.perceptualedge.com/blog/?p=1671]] by S. Few
**[[From Giant Hairballs to Clear Patterns in Networks|http://www.perceptualedge.com/blog/?m=201305&paged=2]] by S. Few (in praise of Dunne & Shneiderman's //Motif Simplification//)
**<<cite Erickson2013>> on domestic Electricity consumption portal (CHI '13)
!!References
<<bibliography>>
*Meetings in June: ''06/11'', 06/18, 06/25
1. Task Taxonomy Revisions
*see reviews and notes (doc), also in SVN with inline comments in ~TeX file
*planned revisions TBD
*recently read: [[notes|Task Characterization: Meta Analysis]]:
**<<cite Lammarsch2012 bibliography:Bibliography>>: connecting "what" to "why" (sort of)
**<<cite Raskin2000>>: p.104 is relevant and worth citing in Table 1
**<<cite Andrienko2006>>: this book is dense and at times amusing for the wrong reasons:
>//"Ქ not going to undertake a detailed analysis of every existing taxonomy in order to reveal its weaknesses in comparison with our superb task typology."//
**<<cite Aigner2011a>>: it simplifies <<cite Andrienko2006>>'s task typology immensly
**<<cite Ware2013>>: visual thinking algorithms
**<<cite Dork2011>>: the "information flaneur"
**<<cite Dork2012>>: search / browse / visualize / stroll
2. Pulse Energy
*Iteration on Mitacs Application
*Granted access to Pulse Energy Manager tool for UBC
*email introduction to Lillian Zaremba (UBC Energy Manager) - to arrange a meeting
*upcoming: energy Manager introductions at ~UVic, ~McGill, UC Berkeley
*did not receive permission to attend ~Pulse-FortisBC meeting (06/07)
3. Overview
*recently read: <<cite Liu2013>> on evaluating a tool for analyzing clusters in large document collections using Newdle
*implications for CHI paper in the wake of ~MoDisco v2 VAST rejection?
*other potential venues: GI '14, AVI '14 (Dec. deadlines)
*to resume after Task Taxonomy revisions
4. Thesis Proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*aligning with CHI doctoral consortium (6 page extended abstract)
*to begin after Task Taxonomy revisions
5. Internship search (long-term)
*Bloomberg has a new research internship program beginning this summer (including UX research); will be seeking applicants for next summer in the fall - to follow-up w/ RR and K. Lau
6. Misc.
*DRITW revisions (MB token until mid this week) - implications for DRITW if Task Taxonomy is accepted
*M.Sc follow-up TOCHI journal paper (w/ CT) rewrite, possible return to data analysis (found an error in how the timing log was implemented)
**TBD w/ JM + CT 06/13
*Released [[ShinyFork|http://spark.rstudio.com/mattbrehmer/ShinyFork/]], a tool for exploring music reviews, written in R/Shiny/ggplot2
!!References
<<bibliography>>
*Meetings in June: 06/11, ''06/25''
1. Task Taxonomy Revisions
*see revisions and cover letter (SVN / email)
*recently read: [[notes|Task Characterization: Meta Analysis]]:
**<<cite Vicente1999 bibliography:Bibliography>> on [[Cognitive Work Analysis]]
**<<cite Smith2002>>, <<cite Bailey1994>> on typologies vs. taxonomies
2. Pulse Energy
*Iteration on Mitacs Application - KT reply 06/24 - suggestion to push back start date to Nov. / or mid Oct
**data collection / interviews to be longer than 1 week (preceding start date)
**TBD: ethics, budget
*Granted access to [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]]
*email introduction to Lillian Zaremba (UBC Energy Manager) - to arrange a meeting
*upcoming: energy Manager introductions at ~UVic, ~McGill, UC Berkeley
3. Overview
*recently read: <<cite Liu2013>> on evaluating a tool for analyzing clusters in large document collections using Newdle
*implications for CHI paper in the wake of ~MoDisco v2 VAST rejection?
*other potential venues: GI '14, AVI '14 (Dec. deadlines)
*to resume after Task Taxonomy revisions
4. Thesis Proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*aligning with CHI doctoral consortium (6 page extended abstract)
*to begin after Task Taxonomy revisions
!!References
<<bibliography>>
*Meetings in July: ''07/02'', 07/09, 07/16, 07/23, 07/30
1. Pulse Energy
*Iteration on Mitacs Application - sent to Mitacs Business Development director Sang Mah (prime BD for Computer Science) for feedback
*Granted access to [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]]
**[[interface analysis|Pulse Energy Manager Notes]]
*email introduction to Lillian Zaremba (UBC Energy Manager) - to arrange a meeting
**using the task typology as a lens for analysis of tasks
**upcoming: energy Manager introductions at ~UVic, ~McGill, UC Berkeley
2. Overview
*recently read: <<cite Liu2013 bibliography:Bibliography>> on evaluating a tool for analyzing clusters in large document collections using Newdle
*Analyzing data with the lens of the task typology
3. Thesis Proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
*Timeline: to occur by end of summer 2013 (according the the [[grad handbook|https://www.cs.ubc.ca/students/grad/policies/grad-handbook]])
*aligning with [[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]]
**4p extended abstract + 1䡴ement of expected benefit, letter of nomination from TM, current CV
4. Misc.
*Attended meetup 06/26: [[Analyzing User Behavior at Plenty of Fish: Data Science in the Wild|http://meetu.ps/1C1HbH]] - Vancouver R User Group / Vancouver ML / Data Science
*Attending meetup 07/02: [[Two Sessions: Telling Lies With Maps and Visual Analytics in Video Game Design|http://meetu.ps/1Nk56H]] - Vancouver Data Visualization
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray
*reading:
**<<cite Vicente1999>> on cognitive work analysis: [[notes|Evaluation in HCI: Meta-Analysis]] - implications for future research / DSM
**<<cite Bailey1994>> on classification: taxonomy vs. typology
**[[How to Report an F Statistic|http://www.yorku.ca/mack/RN-HowToReportAnFStatistic.html]] by I. S. Mackenzie, York U
!!References
<<bibliography>>
*Meetings in July: 07/02, ''07/09'', 07/16, 07/23, 07/30
1. Pulse Energy
*Iteration on Mitacs Application
**sent to Mitacs Business Development director Sang Mah (prime BD for Computer Science) for feedback (round 2 + ORS procedure clarification - no response from SM since 07/03 - I will ping Sang if I don't hear back by tomorrow)
*Granted access to [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]]
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity in terms of //how// and //what//, but not //why// (task analysis will put all of this together)
*LZ (UBC Energy Manager)
**meeting with her 07/09 1pm at her office, open-ended contextual interview re: job duties (work analysis), current usage of Energy Manager, limitations
*to do:
**energy Manager introductions at ~UVic, ~McGill, UC Berkeley - awaiting reply from KT w/ contact information (no reply since 06.28, I will him if I don't hear back from him by tomorrow)
**task typology as a lens for analysis of tasks
2. Overview
*last week: finished on a hybrid transcript / notes document for the MK interview: [[Overview - MK interview transcript/notes]] (Feb 2013)
*other interview notes/transcripts: [[Overview - JG interview notes]] (June 2012), [[Overview - JW interview transcript]] (Dec 2012)
*recently read: <<cite Liu2013 bibliography:Bibliography>> on evaluating a tool for analyzing clusters in large document collections using Newdle
*to do:
**analyzing data with the lens of the task typology
**touch base w/ JS, find out if user JW used web-Overview for another story
**pre-paper talk - TM + SI meeting in late July / early August?
3. Thesis Proposal
*current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
**Timeline: to occur by end of summer 2014 (according an email from Joyce last week, not August 2013 as originally thought)
*aligning with [[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]]
**4p extended abstract + 1䡴ement of expected benefit, letter of nomination from TM, current CV
4. Misc.
*~InfoVis final acceptances TBA 07/11
*DRITW token comes back to me 07/15
*Attending meetup 07/02: [[Two Sessions: Telling Lies With Maps and Visual Analytics in Video Game Design|http://meetu.ps/1Nk56H]] - Vancouver Data Visualization
**(not terribly engaging or enlightening, a low bar?)
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray
**(mostly done, made it through 10/12 chapters quite quickly)
*TOCHI paper w/ CT, JM - meeting 07/10 2pm to discuss plan of action / revisions
!!References
<<bibliography>>
*Meetings in July: 07/02, ''07/16'', 07/25
1. Pulse Energy
*Iteration on Mitacs Application
**sent to Mitacs Business Development director Sang Mah (prime BD for Computer Science) for feedback (round 2 + ORS procedure clarification - no response from SM since 07/03 - pinged again 07/10, got an out-of-office reply) - proceed to get signatures anyway?
**to do: email SM, cc TM, re: [[ORS signature page|http://www.ors.ubc.ca/sites/research.ubc.ca/files/uploads/documents/ORS/Signature%20page%20revised.doc]], request proceed to get signatures - done (ORS contact M. Kirk)
*Granted access to [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]]
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity in terms of //how// and //what//, but not //why// (task analysis will put all of this together); *partially done*
*met w/ [[LZ, UBC ECM|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]] last week; @@color:#bb0000; ''TM to do''@@: skim/read [[meeting + documentation notes|Pulse-LZ-13.07.09]]
*to do - Energy manager introductions/setting up meetings:
**07/15 - KT sent ~McGill contact (made introduction, but didn't cc me)
**07/09 - LZ sent UBC BSM Ops. and SES consulting contacts
**KT to introduce energy managers at ~UVic, UC Berkeley
*to do: email KT re: Mitacs signature + meet to discuss summer/fall Pulse rollout, whether process fixes will occur (done - Mon Jul 22)
2. Task Typology
*2nd round review comments:
**Q1: 5 terminology points: levels of specificity, //discover//, //produce//, //change//, //annotate// vs. //record//
***to do: fix fig. 1 to resolve levels of specificity, //discover// (done); if possible, rephrase produce //produce//, //change//, //annotate// vs. //record// points
**Q2-4 resolved
**minor points: 毲matting, Vicente why-what-how, table 1 row colours (done)
***to do: add corresponding colours to table 1, add Vicente comment, remove DRITW ref (done)
**formatting/style requirements verified
*No facility on PCS (yet) to upload camera-ready version for Aug. 1
***to do: cover letter, prepare camera-ready
*[[new pub page|http://www.cs.ubc.ca/labs/imager/tr/2013/MultiLevelTaskTypology/]] to do: add link from infovis group page - asked permission from JD
3. DRITW
*Token from MS -> SI, MB (July 15); task typology paper and corresponding edits to DRITW manuscript
**read version diffs, cover letter
**to do: proofread .tex, supplemental materials; email SI re: classification/regression sentence ("classification" is overloaded; need to convince R3 - nitpicky?); Kandogan/Ankerst ref
4. Overview
*recently finished on a hybrid transcript / notes document for the MK interview: [[Overview - MK interview transcript/notes]] (Feb 2013)
*other interview notes/transcripts: [[Overview - JG interview notes]] (June 2012), [[Overview - JW interview transcript]] (Dec 2012)
*recently read: <<cite Liu2013 bibliography:Bibliography>> on evaluating a tool for analyzing clusters in large document collections using Newdle
*to do:
**analyzing data with the lens of the task typology
**touch base w/ JS, find out if user JW used web-Overview for another story
**pre-paper talk - TM + SI meeting in late July / early August?
5. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions
**current working title: //~Multi-Level Task Analysis in the Design and Evaluation of Visualization for Data Exploration//
**4p extended abstract + 1䡴ement of expected benefit, letter of nomination from TM, current CV
6. Misc.
*~InfoVis group meeting to do: group editing access to group webpage; party?
*Attending meetup 07/02: [[Two Sessions: Telling Lies With Maps and Visual Analytics in Video Game Design|http://meetu.ps/1Nk56H]] - Vancouver Data Visualization
**(not terribly engaging or enlightening, a low bar?)
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray
**(mostly done, made it through 10/12 chapters quite quickly)
*[[cs.ubc.ca/~brehmer website|http://www.cs.ubc.ca/~brehmer/]] redesign using group/infovis as template
*TOCHI paper w/ CT, JM - ongoing draft revisions until 07/31/13, then token to CT
!!References
<<bibliography>>
*Meetings in July: 07/02, 07/16, ''07/25''
1. Pulse Energy
*Mitacs application submitted
*Granted access to in-dev demo version of [[Pulse Energy Manager tool for UBC|https://ca.pulseenergy.com]] (13.07.22)
**[[interface analysis|Pulse Energy Manager Notes]] - breaking down the interface and interactivity in terms of //how// and //what//, but not //why// (task analysis will put all of this together); *partially done*
*met w/ [[LZ, UBC ECM|http://sbsp.ubc.ca/2012/02/28/lillian-zaremba/]] 13.07.09: [[meeting + documentation notes|Pulse-LZ-13.07.09]]
*meeting w/ JC, ~McGill energy manager 13.07.29
**07/24 - KT sent UC Berkeley energy manager contact - available in August for interview
**07/09 - LZ sent UBC BSM Ops. and SES consulting contacts
**KT to introduce energy managers at ~UVic, non-university energy managers
2. Task Typology
*@@color:#bb0000; ''TM to do''@@: read camera-ready cover letter, view camera ready edits (in SVN, PCS, 13.07.17 email) - due Aug 1
*[[new pub page|http://www.cs.ubc.ca/labs/imager/tr/2013/MultiLevelTaskTypology/]] to do: add link from infovis group page - TM to give group edit permissions
3. Overview
*currently working on pre-pre-paper talk slides
*case study analysis:
**partial transcript / notes document for the MK interview: [[Overview - MK interview transcript/notes]] (Feb 2013)
**other interview notes/transcripts: [[Overview - JG interview notes]] (Dec 2012), [[Overview - JW interview transcript]] (June 2012)
**analyzing data with the lens of the task typology
*to do:
**touch base w/ JS, find out if user JW used web-Overview for another story
**pre-pre-paper talk - TM + SI meeting Aug 1
*meanwhile: JS courting the digital humanities folks by [[Comparing text to data by importing tags|http://overview.ap.org/blog/2013/07/comparing-text-to-data-by-importing-tags/]]
*recently read: <<cite Liu2013 bibliography:Bibliography>> on evaluating a tool for analyzing clusters in large document collections using Newdle
5. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions
**current working title: //~Multi-Level Abstract Task Analysis for Visualization Design and Evaluation//
**4p extended abstract + 1䡴ement of expected benefit, letter of nomination from TM, current CV
6. Misc.
*learning d3 and reading [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345/index.html]] by Scott Murray
*related:
**D3 [[blocks by mbostock|http://bl.ocks.org/mbostock/]]: [[F+C via brushing|http://bl.ocks.org/mbostock/1667367]], [[day/hour heatmap|http://bl.ocks.org/tjdecke/5558084]], [[calendar view|http://bl.ocks.org/mbostock/4063318]]
**[[crossfilter.js|https://github.com/square/crossfilter]] by square
**[[cubism.js|https://github.com/square/cubism]] by square
**[[Trifecta|http://trifacta.com/]] / [[@jheer|https://github.com/jheer]]'s [[vega|https://github.com/trifacta/vega]]
**[[rickshaw.js|http://code.shutterstock.com/rickshaw/]]
*hacking about with [[R+Shiny+GoogleVis|http://cran.r-project.org/web/packages/googleVis/index.html]]
*TOCHI paper w/ CT, JM - ongoing draft revisions until 07/31/13, then token to CT
!!References
<<bibliography>>
*Meetings in September: 09.05, ''09.12'', 09.18 1-3pm
1. Pulse
*Mitacs application approved
**revisions submitted 09.10
*upcoming Pulse Energy Manager user interviews:
**to meet w/ NV, SES, 09.16
**KT to introduce energy managers at Capilano, ~UVic, non-university energy managers (pinged 09/03)
2. Overview
*TM token to JS: feedback / status?
*to ~InfoVis group 09.16 to read, discuss on 09.18, TM + MB debrief afterward
*case studies:
**interviewed SA, Dallas journalist, 08.22 (off-the-record, WIP, to follow-up when completed
**notes: [[Overview - MK interview transcript/notes]] (Feb 2013), [[Overview - JG interview notes]] (Dec 2012), [[Overview - JW interview transcript]] (June 2012)
*B. Shneiderman's comments re: DSM
3. Task Typology
*Registered 09.05 ($360), renewed ACM student membership ($60)
*Flights booked 10.14-18 ($560)
*Accomodations booked (sharing w/ JF @ Best Western)
*VIS video preview submitted
*VIS [[fast forward slides|http://www.cs.toronto.edu/~fchevali/vis2013ffw/pmwiki.php?n=Main.HomePage]]: draft, present to ~InfoVis group 09.16, due 09.24
*~InfoVis group practice talk 10.09 (to do: prepare and rehearse talk)
*Talk likely on afternoon of 10.15
4. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] instructions, submission date 10.04, planning to have draft 10.01-02
**//Mapping Domain Problems to Techniques: Task Abstractions in Visualization Design and Evaluation//: slides
**4p extended abstract + 1atement of expected benefit, **letter of nomination from TM** (reminder to be sent 09.18), current CV
5. Misc.
*--to meet w/ L. Wilkinson 09.12 4pm--
*to meet w/ S. North 09.16 10am
*read JD's IV journal draft, LO's CHI draft
*In Ontario 09.20-30 (Vacation 09.23-27)
*Meetings in October: ''10.03'', 10.10, 10.24, TBA week of 10.29
1. Task Typology
*slide deck outline / draft review
**Schulz and Roth papers
**~InfoVis group practice talk 10.09 (to do: prepare and rehearse talk)
*Benjamin Renoust question re: present / discover
*Talk on afternoon of 10.15
*VIS [[video preview|http://ieeevis.org/year/2013/info/overview-amp-topics/ieee-infovis-papers-30-second-video]] submitted
*VIS FF submitted
*logistics:
**Registered 09.05 ($360), renewed ACM student membership ($60)
**Flights booked 10.14-18 ($560)
**Accommodations booked (sharing w/ JF @ Best Western)
2. CHI '14 Doctoral Consortium
*[[CHI doctoral consortium|http://chi2014.acm.org/authors/doctoral-consortium]] abstract, submission date 10.04
**to do: address JM comments
3. Overview
*on hold until after ~VisWeek; to do:
**consolidate feedback from ~InfoVis group, TM, JS and make revisions, consider [[Survivorship Bias|http://youarenotsosmart.com/2013/05/23/survivorship-bias/]] article
**interview Dallas journalist
**after, token to JS
4. Pulse
*on hold until after ~VisWeek
*met w/ NV, SES, 09.16
*met w/ Pulse's KT 10.02
*start date Oct 28
5. Misc.
*emailed Rock L. at SAP re: potential internship matches for late summer / fall 2014
*DRITW rejection - TBD at ~VisWeek?
*MS ~InfoVis practice talk tomorrow
*Meetings in November: ''11.06'', 11.13, 11.20, 11.27
1. Overview
*JS' student study design document (email link to Google Doc)
**Tried Jigsaw over the weekend
*~EuroVis paper:
**log files sent by SA (Texas); access to documents (need Document Cloud account), tags
**new potential user for interview: TD @ ~ReportersLab
**to do: factor in revisions based on paper draft comments
*CPSC 344/544 Guest lecture (see below)
2. [[Pulse|Pulse Energy Manager]]
*recently: interviewed [[JC (Pulse) 11.04|Pulse-JC-KT-13.11.04]], [[MT (Surrey SB) 10.24|Pulse-MT-13.10.24]], [[NV (SES) 09.18|Pulse-SES-NV-13.09.18]]
**interview analysis and user requirements document: see slide deck in SVN, to present to KT tomorrow
*gave an informal talk about my research / background / UBC ~InfoVis 11.01
3. Misc.
*I'm giving a 35 min guest lecture in CPSC 344/544 Nov 14 (9:30 am): "//Where HCI meets Visual Analytics, Document Leaks, and Investigative Journalism//", abstract:
>Over the past few years, there have been a number of high-profile leaks of sensitive collections documents, such as the leak of hundreds of thousands of top-secret US State Department diplomatic cables to ~WikiLeaks in 2011. When investigative journalists gain access to these document collections, which are often comprised of unstructured text, it's hard for them to know where to begin: a document collection might contain several newsworthy stories, but finding them is difficult through keyword search alone. Since 2010, the UBC Information Visualization research group has collaborated with the Associated Press, the Columbia Journalism School, and a number of investigative journalists toward the design and evaluation of Overview (http://overviewproject.org), a web-based visual analytics application for journalists that allows them to systematically explore large and messy text document collections, helping them to find recurring patterns and anomalous documents. I'll talk about where HCI fits into this ongoing project, addressing questions such as: (1) What makes a visual analytics application effective? (2) What are some of the challenges of working with external collaborators and subject matter experts? (3) How do you study user adoption after you've released an application to the web? and (4) How do you design for infrequent or one-time use by users with a range of technical backgrounds?
*CHI review submitted
*upcoming meetups:
**[[Data Science:Exploratory Analysis, Machine Learning, Statistics, Prediction and Visualization|http://www.meetup.com/DataScience/events/149038502/]] (Tu) 13.11.26 18h30, @HootSuite
***Elena Popovici: //"Small Data, Rich Data: Exploratory Analysis and Visualization"//
*** Kazem Jahanbakhsh: //"How I Used Machine Learning & Statistics To Predict The US Presidential Election"//
!!References:
<<bibliography>>
*Meetings in November: 11.06, ''11.13'', 11.20, 11.27
1. Overview
*JS' student study design document
**see copy of email (below)
*~EuroVis paper:
**log files sent by SA (Texas); granted ~DocumentCloud account have access to Overview projects but not document texts, tags
**new potential user for interview: TD @ ~ReportersLab
**ongoing: revisions based on paper draft comments, new timeline figure
**re-reading: Kang/Stasko VAST '12 paper on Jigsaw case studies: [[VisWeek12 Notes]]
*CPSC 344/544 Guest lecture tomorrow: "//Where HCI meets Visual Analytics, Document Leaks, and Investigative Journalism//"
2. Pulse
*task and data abstraction analysis: slides and spreadsheet in SVN
*this week:
**attending meetings with analytics group
**becoming familiar with Pulse API (R packages for API access, ~MySQL Workbench), organization and space metadata, point data
**to do: 2nd round interviews with UCB, ~McGill energy managers (portfolio users); waiting on response from BC Retirement Concepts energy manager
*TBD: Pulse is strict about where data is stored for Can/US clients; Where will prototypes be hosted?
*~InfoVis group meeting on data/task abstractions? Nov 20? 27?
3. Misc.
*UWO Sedig et al.
*upcoming meetups:
**[[Data Science:Exploratory Analysis, Machine Learning, Statistics, Prediction and Visualization|http://www.meetup.com/DataScience/events/149038502/]] (Tu) 13.11.26 18h30, @HootSuite
***Elena Popovici: //"Small Data, Rich Data: Exploratory Analysis and Visualization"//
*** Kazem Jahanbakhsh: //"How I Used Machine Learning & Statistics To Predict The US Presidential Election"//
!!Overview student study discussion - 13-11-09
!!!Timeline
What are the conditions of your funding? You've mentioned that the study has to be run in Spring 2014. Are you required to have results analyzed and a research paper submitted by the end of Spring 2014 as well?
!!!Ethics
Neither of us have experience conducting studies which involve a possible impact of experimental condition on students' grades. We will investigate the procedures re: ethics proposals for this type of study. We know of some researchers in our department who are doing pedagogical research on technology in the classroom and have faced this issue in the past. You'll want to inquire about precedents for studies of this kind in your school as well. For instance, students should be informed that their grade is based on their story alone, and that any other data collected from them (such as time spent performing analysis or their raw notes) will not be shared with the instructor or expert reviewers and will not affect their grade. We hypothesize that students in a [read-only] or [read + search] condition may be at a disadvantage, and students will also be aware of this disadvantage and this awareness could lead to potential anxiety about their grade.
!!!Focus
Your original idea of comparing [read only] vs. [read + search] vs. [read + search + vis], a progression of increasing functionality, could allow for a richer qualitative comparison than a largely quantitative comparison between [other tool] vs [read + search] vs. [read + search + tagging + viz]. The latter design may tell us which is faster or "better", as measured by # of insights or expert review scores, while the former design could tell us more about how the conditions differ, and why a certain approach or workflow was undertaken given the constraints of the condition. Another possible set of conditions to consider is: [read + search] vs. [read + search + tagging] vs. [read + search + tagging + viz].
!!!Dependent measures
We should attempt to gather quantitative results but not rely on them. We should place a higher priority on qualitative results because we are more certain that they can be acquired. Some concerns about quantitative measures include: (1) timing results based on interface logging will only give us a coarse indication of how many documents were viewed and how many cluster nodes were expanded, but not of how many were actually read, or how much time was spent consulting other sources. (2) we shouldn't expect that everyone will record all of their "insights" in a dedicated notepad panel, though we can qualitatively compare what is written in here. (3) Discrete insight quantification is unlikely, and we should view comprehensiveness as a qualitative measure rather than as a discrete quantitative measure. Comprehensiveness shouldn't be viewed as good or bad but as a qualitative distinction, especially in the case of well-written, well-reported "smoking gun" story that focuses on a small portion of a document set. Discrete insight classification may be more plausible in the natural sciences, such as Saraiya and North's work in bioinformatics contexts, than in journalism/document set analysis. (4) A qualitative comparison will allow us to accommodate participants who defy expectations about their condition, such as those in the [+ viz] condition who rely solely on search, or those in the [read + search + tagging] condition who choose not to tag, whereas we would have to throw out this data in a quantitative analysis.
!!!Questionnaires and sampled interviewing
We suggest pre- and post-hoc questionnaires about workflow and analysis strategy. Daily structured diary questionnaires would also be nice but we shouldn't expect everyone to do them. We could send automated emails with a link to a survey for every day of the study, and perhaps offer some incentive (gift card or a draw?) to fill them out. Following the study, it's unlikely that we'll be able to do detailed interviews with all ~36-45 participants, but a sampled approach in which we interview 1-2 participants from each of the 9 conditions could provide some rich data about workflows and strategies.
!!!The [other tool] condition
Jigsaw is not ready for prime-time. The learning curve is substantial (a 30p "intro to Jigsaw" tutorial PDF) and the tool suffers from many of the same issues that we observed with early versions of Overview: multi-step installation and data pre-processing, explicit saving of project files and workspace files, and no indication of computation progress. I didn't have much of a problem with their demo dataset (a few hundred paper abstracts), but when I tried loading my own ~15K document set, it only loaded half of them and without any indication of error. Document clustering has to be done explicitly from a menu command and asks users to set the value of k (default k = 20). I set it to 10 and then executed the command to cluster䨩s just about crashed my Macbook Air. There was never any indication of clustering progress. It appears that a clustering is saved with a jigsaw project file, but we would need to perform this computation on a more powerful machine.
Jigsaw's "tablet view" for note-taking and evidence gathering is not a text editor but a canvas diagram tool, where the user draws connections and timelines between textual notes and "bookmark" screenshots from the other views. Apparently the tablet view state is saved with a workspace file, but this appears to be really buggy; it seems to save/load correctly one time but not the next. Jigsaw might eventually work for those willing to sink the time into learning the tool, likely full-time text analysts in intelligence settings, but as we've found in our current journalist interview study, a journalist might infrequently use a tool like Overview, perhaps only once, and they can't wait overnight for documents to cluster.
I think we know enough to make a qualitative comparison of Overview to Jigsaw in our current paper. If you still want to pursue a comparison against another tool in the student study, perhaps we can pursue another commercial tool, perhaps NUIX or legal e-discovery software.
*Meetings in November: 11.06, 11.13, ''11.20'', 11.27
1. Overview
*JS' student study design document
**JS' response to my previous email (See email)
*~EuroVis paper:
**consent to discuss story anonymously granted by SA (Dallas), we are permitted to acknowledge Dallas Morning News, but not cite specific story
**new potential user for interview: TD @ ~ReportersLab
**ongoing: revisions (as of 9am 11/20, two pages left on this iteration), timeline:
>Tonight > Fri Nov 22 - TM editing
>Sat Nov 23 Sun Nov 24 - MB editing
>Mon Nov 25 > Tue Nov 26 - JS editing
>Wed Nov 27 > Sun Dec 1 - back and forth between MB, TM, JS as needed
>Fri Nov 29 > submit abstract by 16h GMT
>Mon Dec. 2 > Tue Dec 3 - Stephen final look-over
>Fri. Dec 6 > submit by 16h GMT
**re-reading: Kang/Stasko VAST '12 paper on Jigsaw case studies: [[VisWeek12 Notes]], G砥t al. 2013 TVCG
2. Pulse
*last week:
**data/task abstractions, scope: multi-attribute hierarchical portfolio ranking, multi-faceted comparison (drill down, roll up), single building comparison
**KT very pleased with data/task abstraction effort, thinks scope might be too large, could focus on ranking alone
**tags possible, operating hours (open-closed) detectable, base temperature detectable, tags by # occupant and sq. footage less likely
*this week:
*design proposal document for prioritizing ideas (data/task abstractions translated back into domain language)
*data wrangling in R (Pulse API, Data Tables packages) and experimenting with Gratzl et al's ~LineUp
**: 2nd round interview w/ Pulse's JC, ~McGill energy manager (portfolio user); waiting on response from UCB energy manager (2nd round), BC Retirement Concepts energy manager (1st round)
3. Misc.
*gave a talk about Overview to CPSC 344/544 last week; approached by ugrad student interested in getting involved in HCI research
*upcoming meetups:
**[[Data Science:Exploratory Analysis, Machine Learning, Statistics, Prediction and Visualization|http://www.meetup.com/DataScience/events/149038502/]] (Tu) 13.11.26 18h30, @HootSuite
***Elena Popovici: //"Small Data, Rich Data: Exploratory Analysis and Visualization"//
*** Kazem Jahanbakhsh: //"How I Used Machine Learning & Statistics To Predict The US Presidential Election"//
*Meetings in November: 11.06, 11.13, 11.20, ''11.27''
1. Overview
*~EuroVis paper:
**reviewing JS' edits (in SVN), email comments
**reviewing keywords on [[PCS|https://precisionconference.com/~eurovis14/]]
**response from DALLAS re: estimated time without Overview:
>//How long would it have taken me? I would guess about 3-4 times as long. Here's why://
>
>//I ended up putting from 30-70 pdfs in each document set. I have no idea, really, if individual emails were sensibly grouped within those sets when they were provided to me. In other words, I have no idea whether the governmental body put a single, lengthy email chain in a single pdf or spread it over more than one pdf. I entered each page of each pdf individually in Overview, and Overview then grouped them. I think I remember that some of the nodes held pages from different pdfs. Without Overview, single topics would have been scattered over multiple pdfs and would be hard to keep track of.//
>
>//If I had to do this "by hand" I probably would have spent more than a week on this to do the same level of looking. I would have used sticky notes to highlight interesting pages if I had printed it all out, which would have taken time too. If I had read the pdf's on screen, I would have had to devise an whole different system to tag interesting documents, something that makes my brain hurt just contemplating it.//
>
>//I'm definitely planning on using this again for large document sets. My only disappointment about this system is that if I were on a tight deadline, I couldn't know for sure that Doc Cloud would load the documents quickly. Sometimes it takes overnight, and while we usually can handle this, we can't always.//
>
>//All in all, a great tool for journalists.//
**remaining timeline:
>Wed Nov 27 > Sun Dec 1 - back and forth between MB, TM, JS as needed, JS in China for a week-long workshop, leaving Friday
>Fri Nov 29 > submit abstract by 16h GMT
>Mon Dec. 2 > Tue Dec 3 - Stephen final look-over
>Fri. Dec 6 > submit by 16h GMT
**draft reading, who / when: JD, JM, MUX, Guiseppe's group?
*Student study - on hold
2. Pulse
*data/task abstractions and project scope:
**''portfolio level'': rank groups of spaces based on multiple measures of performance, sub-rank within groups. Compare changes in rank over time (#1 priority by KT, JC)
**''portfolio / detail level'': multi-faceted comparison of portfolio performance over time. (#1 priority by ~JC-McGill)
***''drill-down'': split portfolio into spaces OR groups of spaces.
***''שּׁ-up'': determine contribution of spaces OR groups of spaces to overall portfolio performance.
**--''space level'': compare a single spaceథrformance over time.--
*2nd round interview w/ Pulse's JC, ~McGill energy manager (portfolio user) last Friday, different prioritization re: tasks than Pulse's JC; waiting on response from UCB energy manager (2nd round), BC Retirement Concepts energy manager (1st round)
*this week: data munging with Pulse API, Data Tables, meetings with Pulse energy manager product manager, analytics team
*Pulse design brainstorming w/ ~InfoVis group Dec 11 / 18?
3. Misc.
*meetup yesterday:
**[[Data Science:Exploratory Analysis, Machine Learning, Statistics, Prediction and Visualization|http://www.meetup.com/DataScience/events/149038502/]] (Tu) 13.11.26 18h30, @HootSuite
***Elena Popovici: //"Small Data, Rich Data: Exploratory Analysis and Visualization"//: [[Shiny+GoogleVis EDA demos|http://www.icohealthanalytics.com/#!data-visualization/c173s]], case study re: Medicare diagnoses and discharges, following [[Jeff Leek's Data Analysis|https://www.coursera.org/course/dataanalysis]] framework
***Kazem Jahanbakhsh: //"How I Used Machine Learning & Statistics To Predict The US Presidential Election"//: sentiment analysis + LDA
*work on TOCHI paper w/ CT, JM resuming Dec 11: video coding, data wrangling and analysis, paper revisions
*Meetings in December: ''12.04'', 12.11 (off campus?), 12.18 (off campus?)
1. Overview
*~EuroVis paper:
**reviewing TM edits + JD comments
**reviewing submission on [[PCS|https://precisionconference.com/~eurovis14/]]
**TBD:
***Should we anonymize reference to Stephen's dissertation?
***Abstract:
>//"We choose an ambitious target to measure the success of our task abstraction"// (sequence of events?)
***RW: Chen et al's system is called "Exemplar-based visualization", not "Exemplar"
***Discussion:
>//"One difference from most prior systems is that Overview uses scalable, linear-time algorithms; although none of our case studies involved over 10,000 documents, the motivating iraqlogs collection has almost 400,000 documents."// (pre-processing time?)
***Case studies: should we consistently refer to them with //"CS#"//? should these be in the timeline figure?
***Acknowledgments: anonymize?
**schedule:
>Wed. Dec 4 > MB edits
> Thu. Dec 5 > SI edits?
>Fri. Dec 6 > submit by 16h GMT
*Student study - on hold
2. Pulse
*data/task abstractions and project scope:
**''portfolio level'': rank groups of spaces based on multiple measures of performance, sub-rank within groups. Compare changes in rank over time (#1 priority by KT, JC)
**''portfolio / detail level'': multi-faceted comparison of portfolio performance over time. (#1 priority by ~JC-McGill)
***''drill-down'': split portfolio into spaces OR groups of spaces.
***''שּׁ-up'': determine contribution of spaces OR groups of spaces to overall portfolio performance.
**--''space level'': compare a single spaceథrformance over time.-- (consolidated as being special case of "drill-down" for single space)
*meeting UCB energy manager (2nd round) Friday, KT to set up meeting w/ British Utility Level User
*last week: met w/ energy manager product team manager, met with analytics developer to discuss data formatting, glorious capabilities of R's data.table package
*this week: developing "Portfolio Sandbox" visualization prototyping environment (with R / Shiny / ggplot2, data.table), to meet with DH (Pulse CEO) Friday
3. Misc.
*GRAC work begins귯rk on TOCHI paper w/ CT, JM resuming Dec 11: video coding, data wrangling and analysis, paper revisions
*Meetings in December: 12.04, ''12.11 (@ Wicked Caf秬 12.18 (@ Wicked Caf걮 Pulse
*data/task abstractions and scope:
**''portfolio level'': rank groups of spaces based on multiple measures of performance, sub-rank within groups. Compare changes in rank over time
**''portfolio / detail level'': multi-faceted comparison of portfolio performance over time.
***''drill-down'': split portfolio into spaces OR groups of spaces.
***''שּׁ-up'': determine contribution of spaces OR groups of spaces to overall portfolio performance.
**''TM'' Q: do utility-level analysis ever do forensic single-space analysis?
**understanding the workflow from portfolio-level analysis to portfolio-detail analysis (after feedback from JC (~McGill) and KN (UCB))
*progress on "portfolio sandbox", R/Shiny/ggplot prototyping platform for portfolio-level analysis and portfolio-detail level analysis; since last week:
**normalization, aggregation, heatmaps, bumpcharts, boxplots, faceting, data munging, code-refactoring
**''TM'': normalizing box plots and omitting outliers with some visual indicator of outliers' presence; alternative ways of showing uncertainty and ranges 졠Vismon (<<cite Booshehrian2012 bibliography:Bibliography>>: heatmap boxplot hybrids); adjusting dynamic range within and between cells / scale of boxplots to faciliate comparisons; not showing absolute scale by default, but relative and without plotting outliers
**''TM'': hue families for stacked bars (multiple resources); using Pulse colour scheme, using position for time comparison, not colour;
*a summary of what was discussed yesterday w/ KT regarding next steps in terms of exploring and prototyping visualizations for portfolio-level analysis. These items are sorted from big-picture ill-defined design problems to more concrete and well-defined design ideas
**understanding the workflow from portfolio-level analysis to portfolio-detail analysis, (also: gathering more user data by speaking to utility company users)
**design implications for tagging spaces (awaiting space-tag lists from JC (~McGill) and KN (UCB))
** indicators of missing data (# records), standard deviation, and ranges in aggregated data (as you saw, boxplots don't scale)
**approaches to multi-attribute weighted rankings (졠[[LineUp|lineup.caleydo.org]]), flexibility vs. defined meaningful weightings
**coordinated bar and bump charts (also 졠[[LineUp|lineup.caleydo.org]]; Ikely hitting a wall with Rඩsualization package options here, given that this is a fairly novel visual encoding combination; I may need to prototype in a different language environment to explore designs in this area).
**HDD/CDD normalization: determine which spaces in a portfolio are correlated with ~HDDs/~CDDs, subsetting / faceting by these
**coordinated / aligned weather charts across time facets
** differential charts: comparing absolute and relative differentials of this year:last year or actual:baseline; use of [[horizon charts|[[horizon charts|http://timelyportfolio.blogspot.ca/2012/08/horizon-on-ggplot2.html]] or heat maps
**interactive toolips / higlighting and selection, details-on-demand, alternatives to raw ggplot: assessing viability of [[rCharts|http://ramnathv.github.io/rCharts/]] (with [[rickshaw.js|http://code.shutterstock.com/rickshaw/]] and [[leaflet|http://leafletjs.com/]] integration) , [[rVega|https://github.com/metagraf/rVega]] (R wrapper for [[Trifacta|http://www.trifacta.com/]]'s [[vega|https://github.com/trifacta/vega]]), [[rHighcharts|https://github.com/metagraf/rHighcharts]] (R wrapper for [[highcharts.js|http://www.highcharts.com/]], toolkit currently used in Pulse Energy Manager for interactive time-series visualization), [[RStudio|https://github.com/rstudio/rstudio]]'s [[ggvis|https://github.com/rstudio/ggvis]] (ggplot + vega + shiny)
**''TM'': faceting as subset of embedding
**''TM'': scalability: limits of pixels > limits of cognition
*last week: met w/ UCB energy manager (2nd round) Friday, also met and demoed [[LineUp|lineup.caleydo.org]] and "portfolio sandbox" w/ DH (Pulse CEO)
*this week: met w/ KT yesterday to discuss next steps re: "portfolio sandbox" prototyping and integrating feedback from JC (~McGill) and KN (UCB); to meet with SJ, Pulse's product manager tomorrow, demoing [[LineUp|lineup.caleydo.org]] and "portfolio sandbox" in bi-weekly demo session on Friday
*upcoming: KT to set up meeting w/ British (and US?) Utility Level Users
2. Overview
*~EuroVis paper:
**submitted! first round review notification: Feb 14
**forgot to anonymnize citations IRAQLOGS [St10], and IRAQSEC [St12] in timeline figure (should be [Ano10], [Ano12]), citations were hard-coded in figure and not synced to .bib
*Student study - to do:
**to do: email JS re: student study follow-up, email Jigsaw team re: scalability
*new story w/ Overview re: food stamp website: [[Records: DHHS downplayed food stamp issues|http://www.wral.com/records-dhhs-downplayed-food-stamp-glitches/13173174/]] - WRAL.com (Dec 9)
3. Misc.
*CHI DC: notification delayed to as late as Dec 20 (was supposed to be Dec 10, citing high number of submissions) (also applied to CHI SV program)
*GRAC work begins귯rk on TOCHI paper w/ CT, JM resuming Dec 11,18. definitely: paper revisions; maybe: video coding, data wrangling and analysis
!!References:
<<bibliography>>
*Meetings in December: 12.04, 12.11 (@ Wicked Caf젧'12.18 (@ Wicked Caf秊1. Pulse
*data/task abstractions and scope:
**''portfolio level'': rank groups of spaces based on multiple measures of performance, sub-rank within groups. Compare changes in rank over time
**''portfolio / detail level'': multi-faceted comparison of portfolio performance over time.
***''drill-down'': split portfolio into spaces OR groups of spaces.
***''שּׁ-up'': determine contribution of spaces OR groups of spaces to overall portfolio performance.
**''TM'' Q: do utility-level analysis ever do forensic single-space analysis?
***''KN'': Yes.
**understanding the workflow from portfolio-level analysis to portfolio-detail analysis (after feedback from JC (~McGill) and KN (UCB)) (hope to speak to utility company energy analsyts in January)
*last week: met with SJ, Pulse's product manager tomorrow (discused weather normalization, "time-over-time" analysis, single-space level analysis; drill-down from portfolio to single; demoed [[LineUp|lineup.caleydo.org]] and "portfolio visualization sketches" in bi-weekly demo session on Friday
*this week: met w/ KT yesterday to discuss next steps re: "portfolio visualization sketches" *upcoming: KT to set up meeting w/ British (and US?) Utility Level Users in January
*progress on "portfolio visualization sketches":
**design for tagging spaces based on tag list from JC (~McGill) (awaiting space-tag lists KN (UCB))
** indicators of missing data (# records), standard deviation, and ranges in aggregated data
***''TM'': normalizing box plots and omitting outliers with some visual indicator of outliers' presence; alternative ways of showing uncertainty and ranges 졠Vismon heatmap boxplot hybrids); adjusting dynamic range within and between cells / scale of boxplots to faciliate comparisons; not showing absolute scale by default, but relative and without plotting outliers
**coordinated bar and bump charts (졠[[LineUp|lineup.caleydo.org]])
***Update: still trying to figure this out㵲rently working on: side-by-side bump plot and bar chart with linked highlighting / selection (next step: interactive rCharts rather than ggplot); [[LineUp|lineup.caleydo.org]] Java source code deeply embedded in larger application framework, no documentation.
**HDD/CDD normalization: determine which spaces in a portfolio are correlated with ~HDDs/~CDDs, subsetting / faceting by these
***Update: SJ suggested comparing baseload, or comparing demand at midnight
**differential charts: comparing absolute and relative differentials of this year:last year or actual:baseline; use of [[horizon charts|[[horizon charts|http://timelyportfolio.blogspot.ca/2012/08/horizon-on-ggplot2.html]] or heat maps
***to doꪧ'TM'': hue families for stacked bars (multiple resources); using Pulse colour scheme, using position for time comparison, not colour;
***to doꪣoordinated / aligned weather charts across time facetsꪪUpdate: May not be necessary with HDD/CDD normalization? Either / or? An expectation from Energy Manager. Weather wouldnࢥ scalable for geographically heterogeneous portfolios.ꪡpproaches to multi-attribute weighted rankings (졠[[LineUp|lineup.caleydo.org]]), flexibility vs. defined meaningful weightings
***Update: May not be necessary given attribute normalization options
**interactive toolips / higlighting and selection, details-on-demand, alternatives to raw ggplot: assessing viability of [[rCharts|http://ramnathv.github.io/rCharts/]] (with [[rickshaw.js|http://code.shutterstock.com/rickshaw/]] and [[leaflet|http://leafletjs.com/]] integration) , [[rVega|https://github.com/metagraf/rVega]] (R wrapper for [[Trifacta|http://www.trifacta.com/]]'s [[vega|https://github.com/trifacta/vega]]), [[rHighcharts|https://github.com/metagraf/rHighcharts]] (R wrapper for [[highcharts.js|http://www.highcharts.com/]], toolkit currently used in Pulse Energy Manager for interactive time-series visualization), [[RStudio|https://github.com/rstudio/rstudio]]'s [[ggvis|https://github.com/rstudio/ggvis]] (ggplot + vega + shiny)
**''TM'': faceting as subset of embedding
**''TM'': scalability: limits of pixels > limits of cognition
2. Overview
*new story w/ Overview re: food stamp website: [[Records: DHHS downplayed food stamp issues|http://www.wral.com/records-dhhs-downplayed-food-stamp-glitches/13173174/]] - WRAL.com (Dec 9)
**sent email, awaiting response
3. Misc.
*Task book chapter
*CHI DC: notification delayed to as late as Dec 20 (was supposed to be Dec 10, citing high number of submissions) (also applied to CHI SV program)
*GRAC work begins讯 HCI/Vis NSERC fast-tracked applications)
*~EuroVis reviews x2
*work on TOCHI paper w/ CT, JM resuming Dec 18. definitely: paper revisions; maybe: video coding, data wrangling and analysis
*Meetings in January: 4pm Fridays, ''01.10 (at Arbutus Coffee)'', TBD week of Jan 13, 01.24, 01.31
1. Pulse
*Mitacs pay schedule
*portfolio visualization design update:
**loads accomplished/implemented since Dec 18 (last meeting): see [[slides (summary on slides 3-4)|https://dl.dropboxusercontent.com/u/6397998/14.01.10-portfolio-vis-update.pdf]] and [[Journal 14.01.06 - 14.01.10]], [[Journal 13.12.30 - 14.01.03]], [[Journal 13.12.23 - 13.12.27]]
2. Overview
*new story w/ Overview re: food stamp website: [[Records: DHHS downplayed food stamp issues|http://www.wral.com/records-dhhs-downplayed-food-stamp-glitches/13173174/]] - WRAL.com (Dec 9)
**sent email, awaiting response (perhaps worth trying again?)
*another new story: [[For Their Eyes Only 㡮dra Peddie and Adam Playford, Newsday|http://data.newsday.com/crime/police-misconduct/]] (Dec 2013)
*JS using timeline figure for blog post (also: [[JS' Jan 9 blog post on story types|http://overview.ap.org/blog/2014/01/what-do-journalists-do-with-documents-the-different-kinds-of-document-driven-stories/]])
3. GRAC
*U. Pitt students
*secondary review for USTC student
4. Misc.
*Task book chapter
*CHI DC: rejected Dec 20, no reviews, generic automated email response
*CHI SV: currently sitting very low (260ish) on waitlist
*~EuroVis reviews x2 due Jan 21
*work on TOCHI paper w/ CT, JM resuming Dec 18. definitely: paper revisions; maybe: video coding, data wrangling and analysis
*Meetings in January: 4pm Fridays, 01.10, ''01.24 (at Higher Grounds)'', 01.31 (at Bean Around The Worlds, West Point Grey)
*upcoming TM travel: Feb 2-4, 14-21, Mid Mar; May 26 - June in Europe
1. Pulse
*portfolio visualization feedback update:
**conversations w/ ~McGill's JC, Pulse's JC, HR, KT
**upcoming; conversations with UCB's KN, BG's AC
**design: calendar-based heatmaps
**campuses as stepping stones between single buildings and large utility portfolios; a need for guidance, design of workflows but not constraining (졠Dimstiller), breadcrumbs and backtracking, visualizing aggregates in conjunction with components;
2. Overview
*JS sent notes from interview with user AP
3. GRAC
*making visualization tag list public, sharing w/ TM
4. Misc.
*book (third draft complete)
*to do: schedule thesis proposal defense for spring-summer '14
*to do: respin CHI DC submission as IEEE VIS DC submission (July '14)
*Meetings in January: 4pm Fridays, 01.10, 01.24, ''01.31'' (at Bean Around The Worlds, West Point Grey)
*upcoming TM travel: (see email)
1. Pulse
*portfolio visualization feedback update:
**conversations w/ ~McGill's JC, UCB's KN, BG's AC, Pulse's JC, HR, KT, BC
**communicating sketchy workflows / interactions, in addition to sketchy visual encodings: how many visual encodings in conjunction? how to link, combine, or sequence? See Weaver's body of research, esp. IV London paper); a notation for selecting views 졠Tableau: aggregating, filtering, selecting, designating screen-real-estate for overview and detail views
**calendar heatmaps vs. regular heatmaps: simple change in visual encoding leads to massive gain in understanding
**domain specificity vs. general BI tools (Excel, Tableau)
**rankings and false positives: visual encoding of ~LineUp and ggplot implementation differ in salience of bump plot lines: costs and benefits
**rankings are powerful for early exploration when portfolio familiarity is low; otherwise they are powerful for infrequent presentation of long-term trends
**learning from ~DimStiller re: rigid workflows / workflow guidance; allowing direct queriers and specific drill-down patterns
**how to encode missing data vs. missing usage (shutdowns and startups)
2. Overview
*JS sent notes from interview with user AP (to respond)
3. GRAC
*visualization tag list, offer for TL (emailed DP)
4. Misc.
*book (third draft complete: task why-what-how framework to move up to 2nd chapter)
*ongoing: schedule thesis proposal defense for spring-summer '14
*to do: respin CHI DC submission as IEEE VIS DC submission (July '14)
*Meetings in February: Wednesdays @ Broadway/Granville Blenz, ''2pm 02.12'', 4pm 02.26
*upcoming TM travel: (Barbados, SF, Carolina)
1. Pulse
*portfolio visualization feedback consolidation slides
**box plots: data-driven outlier capping not resource-dependent;
**workflow authoring not visual histories; allowing for revisitation and sharing: bookmark a visualization state as something understandable, url sharing, completely configured states;
**temporal granularity: consider weeks as internal date range granularities, rather than months;
**2nd internship could be deployment field study rather than on-site development internship
**domain specificity vs. general BI tools (Excel, Tableau)
**learning from ~DimStiller re: rigid workflows / workflow guidance; allowing direct queriers and specific drill-down patterns - more literature review needed here
**logistics post Feb 28
2. Overview
*JS sent notes from interview with user AP
*radio silence re: journo student study
*~EuroVis acceptance decision expected Friday Feb 14
3. GRAC
*visualization / HCI people / loose ends
4. Misc.
*book draft progress
*ongoing: schedule thesis proposal defense (pending ~EuroVis decision): Wed Apr 16 10-11 (pending 444 exam schedule) or Wed May 7 10-11
*to do: respin CHI DC submission as IEEE VIS DC submission (July '14)
*Meetings in February: Wednesdays @ Broadway/Granville Blenz, 2pm 02.12, ''4pm 02.26''
*upcoming TM travel: (Feb 27 - Mar 2)
1. Pulse
*portfolio viz design / eval update:
**demoed designs to Accenture last week; this week, meeting w/ 2 internal users + developer assigned to Energy Manager project
**[[KT's response to consolidated user feedback document|Journal 14.02.17 - 14.02.21]] (that slide deck from 2 weeks ago)
**box plot / density plot design: distribution-dependent outlier capping implemented
**heatmap colour-scale glyphs on portfolio map implemented
**stacked area chart for maintaining context to/from small multiple line plots implemented
**filter controls streamlined (along with UI in general)
**lineup plot design; small tweaks intended to reduce false positives
**portfolios of larger populations (e.g. all churches in California, clients of a util co.): choropleth maps?
**[[more comprehensive list of prototyping to-dos|Journal 14.02.24 - 14.02.28]]
**points from our last meeting:
***workflow authoring not visual histories; allowing for revisitation and sharing: bookmark a visualization state as something understandable, url sharing, completely configured states; (requires JS prototyping)
***temporal granularity: consider weeks as internal date range granularities, rather than months; (ongoing)
***2nd internship could be deployment field study rather than on-site development internship (TBD)
***learning from ~DimStiller re: rigid workflows / workflow guidance; allowing direct queries and specific drill-down patterns (more literature review needed here)
**logistics post Feb 28: one day / week (Tuesday or Thursday?), eventually regular meetings with 2 developers assigned to project
2. Overview
*submit ~EuroVis rebuttal? (no due date posted)
*~EuroVis reviews + MB comments (see email or SVN: /ubc/cs/research/imager/project/infovis/overview/overview_svn/papers/)
*the ~EuroVis submission in IEEE VIS format (with enlarged fig. 1 as teaser figure) is less than 8 pages (see SVN)
*submit ~EuroVis reviews and cover page / rebuttal as supplemental?
*include new case study?
*still no word on student study
3. GRAC
*7 HCI offers to date (1 will not accept; 5 invited for visit day)
*visit day Mar 7; 1-on-1 student meetings; lab tour
4. Thesis Proposal
*May 7 10am; backup: May 21 10am (no confirmation yet from JM, RR)
5. Misc.
*book draft progress
*Renoust (U. Bordeaux) dissertation and the task typology
*CHI Apr 26 - May 1; [[some viz papers to be presented|References: CHI2014]]
*IEEE VIS Doctoral Colloquium submission (July '14)
*Meetings in May: Friday, May 2 (@ the Brown Dog on West 10th)
*next meeting TBD? Fri 2pm? May 23, Jun 13, Jul 4, Jul 25, Aug 22?
1. Pulse
*continuing to meet w/ KT and dev team; timeline and possible 2nd internship as post-deployment study, interactive workflow brainstorming and prototyping
*Pulse's KT has read proposal //design perspective// chapter
*invited to speak (10-15min) at Pulse customer webinar May 28 on best practices in Viz and/or viz case studies (general energy utility / marketing audience), loosely related to their [[recent infographic|http://www.pulseenergy.com/infographictwo/]]
*recent refs:
**[[Perin and Fekete on ranking tables (Proc. CHI 2014)|http://hal.inria.fr/docs/00/92/98/44/PDF/atable.pdf]]
**[[Albers and Gleicher on tasks and time-series vis (Proc. CHI 2014)|http://graphics.cs.wisc.edu/Papers/2014/ACG14/affordances-preprint.pdf]]
2. Overview
*[[Reflections on the evolution of the Jigsaw visual analytics system|doi.org/10.1177/1473871613495674]] - G砯 Liu / Stasko (Information Visualization journal, in press)
*[[Hiearchie: Visualization for Hierarchical Topic Models|http://nlp.stanford.edu/events/illvi2014/schedule.html]] - Smith et al, UMD (ACL Workshop on Interactive Language Learning, Visualization, and Interfaces, June 27, Baltimore); contacted author for pre-print, awaiting reply
*Overview's NEWYORK case study story a finalist for Pulitzer
3. Thesis Proposal
*May 7 10am in ICICS 304
*[[proposal|http://goo.gl/ulvi0B]], [[supplemental materials|http://goo.gl/vVl1ls]], [[draft slides|http://goo.gl/8xQq9R]]
*Pulse's KT has read proposal //design perspective// chapter
*send //survey perspective// chapter to MS?
4. VIS DC
*submission deadline May 10 5pm
*to do: draft poster (not required but recommended)
*TM to do: [[letter of support|http://ieeevis.org/year/2014/info/call-participation/doctoral-colloquium]]
5. Eval for VA workshop, May 20-21 @UCSD
*submitted statement, waiting for decision from BF, Purdue
6. TM Book draft v4 / task typology
*why/what/how rejiggering
*[[Renoust (U. Bordeaux) dissertation|http://hal.inria.fr/docs/00/94/23/58/PDF/RENOUST_BENJAMIN_2013.pdf]]
*Saket et al. [[graph visualization task taxonomy|http://arxiv.org/abs/1403.7421]] (tech reports from Arizona); [[2nd eval paper|http://arxiv.org/abs/1404.1911]]
*Pretorius, Purchase, Stasko: [[Tasks for Multivariate Network Analysis|http://link.springer.com/chapter/10.1007/978-3-319-06793-3_5]] in Kerren et al (eds). //Multivariate Network Visualization// (2014)
7. Misc
*GRAC: no more meetings, 2 HCI M.Sc students accepted
*VIVA graduate recruitment: contacted 05/01 by lab coordinator via RR; awaiting more info
*x2 ~InfoVis reviews due May 8
*attended meetups:
**[[Van UE: Exploring the research toolbox: what to use, when and why|http://meetu.ps/2jCj5d]] Apr 29
**[[Data Science / R Users group: Deep Learning for High Performance Time-series Databases: Ted Dunning|http://meetu.ps/2dfHfC]] Apr 24
*check out [[Lyra|http://idl.cs.washington.edu/projects/lyra/]] [[Visualization Design Environment (VDE)|http://idl.cs.washington.edu/projects/lyra/app/]] by UW (Heer at al, to appear at ~EuroVis '14) ([[tutorial|http://vallandingham.me/make_a_barchart_with_lyra.html]])
*Meetings in May: Friday, May 2; ''2pm Thursday, May 15 (@ UBC)''
*next meeting week of June 9?
1. Pulse
*continuing to meet w/ Pulse dev team; timeline and possible 2nd internship as post-deployment study, interactive workflow brainstorming and prototyping (Mondays starting May 26)
*working on d3.js calendar tilemaps + (focus + context) + (linking + brushing)
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager//
2. Thesis Proposal
*formally advanced to candidacy
*Additional debrief?
*Reading Hutchins (1995) //Cognition in the Wild//, discusses D. Marr (1982)'s computational models of vision:
**computational level: why and what; representational level: how; implementation level: how
3. Vis Tasks for DR Data (formerly DRITW)
*re-visiting our last discussions from Sept '13
*TVCG round 1 reviewers' list of Viz tools for DR data; mapping tasks to techniques
*BELIV submission deadline June 30
4. VIS DC
*submitted!
*[[stats on submissions|https://docs.google.com/forms/d/1bJDWpawq9YqT2hbvA7f-ANOm4xX-3tx_oZ0o8GhIpX8/viewanalytics?usp=form_confirm]]
5. Eval for VA workshop, May 20-21 @UCSD
*everything booked and confirmed (gone May 19-23)
*no workshop agenda sent yet (?!)
*prepared some slides about Overview, Pulse
6. TM Book draft v4 / task typology
*Anything further to discuss?
7. Overview
*[[JS's post on document-mining Pulitzer winners and finalists|http://overview.ap.org/blog/2014/05/the-document-mining-pulitzers/]] (05/01, updated 05/06)
8. Misc
*GRAC: no more meetings, 2 HCI M.Sc students accepted
*attended GSS seminar on modeling proportions and count data (05/14)
*Y. Dittrich seminar on action research / software engineering (attending today at 4?)
Meetings in Summer 2014:
*''4pm Wed June 11 @ Grindstone Caf駧
*2pm Fri Jul 4, Jul 25, Aug 22 @ Brown Dog - others?
1. Overview
*Conditionally accepted to ~InfoVis! Revisions and statement of changes due Fri Jun 27 5pm PST
*See reviews + MB comments on reviews (sent via email)
*R2 (secondary) and ~HierarchicalTopics
*Additional revisions:
**citing <<cite Gorg2013c bibliography:Bibliography>>, in press Information Visualization journal article reflecting on Jigsaw (the value of supporting the reading of individual documents, document import is critical)
**citing <<cite Smith2014>> ACL workshop article on [[Hi顲chie|http://decisive-ui.github.io/Hierarchie/#/about]]: [[A structured display of topics from discussion surrounding the MH-370 disappearance|http://decisive-ui.github.io/Hierarchie/#/]]
**correcting two obvious typos that reviewers didn't spot!
**giving explicit figure credit to Lloyd/Dykes
**updating SI's affiliation?
**adding comment re: NEWYORK case study story as 2014 Pulitzer finalist
2. Viz Tasks for DR Data (fka DRITW)
*BELIV pre-paper talk slides (see SVN link, sent via email)
*BELIV submission deadline June 30
*writing schedule, feedback from MS, SI
3. Science of Evaluation Workshop @ UCSD May 20-21
*TBD
*reimbursement process underway
4. Empirical Work to Inform Visualization Design
*a positive outcome of the Science of Evaluation Workshop
*BELIV position paper with SC (Calgary), MT (U.Vic), BL (MSR)
5. Pulse
*working on d3.js [[color stock tilemaps + (focus + context) + (linking + brushing)|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] (challenge: coordinated interactivity / highlighting in d3 with small multiples)
*last 2 weeks: meeting w/ UX designer, KT, dev team; some features coming online this month for new customers only (small multiple tilemaps and boxplots); technical product analyst is planning to conduct interviews, MB to join; tablet design considerations for energy managers conducting site inspections;
*[[original R/Shiny/ggplot implementation of portfolio visualization sandbox|http://spark.rstudio.com/mattbrehmer/ShinyPortfolio/]] (Feb 2014, Pulse-hosted instance no longer functioning due to R version / package version upgrades/conflicts made by Pulse Analytics team)
*CHI '15 paper on design? ~EuroVis '15 paper?
*TBD: 2nd internship as post-deployment analysis? timeline?
*[[blog post from Francis Rowland, UX Designer|http://ebiinterfaces.wordpress.com/2013/03/13/activity-centered-design-some-thoughts/]] on tasks/activities/workflows
**suggested: Vicente's critique of activity theory as a descriptive tool not originally intended as a part of a design practice; //Cognitive Work Analysis// as a formative analysis tool (<<cite Vicente1999>>)
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager//
6. Task Typology
*TM update on Bordeaux collaboration
*<<cite Mirel2014>> [[BMC Bioinformatics article|http://www.biomedcentral.com/1471-2105/15/117/]] uses Typology to convey task analysis (w/ C. G沧)
*Saket et al (U. Arizona)'s [[Group-Level Graph Visualization Taxonomy|http://arxiv.org/pdf/1403.7421v1.pdf]] (short paper) presented at ~EuroVis 2014
*Saket et al (U. Arizona)'s [[Node, Node-Link, and Node-Link-Group Diagrams: An Evaluation|http://arxiv.org/pdf/1404.1911v1.pdf]] conditionally accepted to ~InfoVis 2014
*RC (Tufts U.) a big fan
7. Misc
*upcoming 0.5 MUX slot Jun 18: what to discuss?
*MB (Harvard U) ~PhD Dissertation [[Perception, Cognition, and Eॣtiveness of Visualizations with Applications in Science and Engineering|http://dash.harvard.edu/handle/1/12274335]]
*[[EuroVis '14 design study session features no actual design studies|http://eurovis.swansea.ac.uk/program.htm]]
*check out [[setviz.net|http://www.setviz.net/]] by Alsallakh et al and other ~EuroVis STARS (State of the Art Reports)
*coming soon: [[p5.js|http://p5js.org/]], a [[new JS client-side library|https://github.com/lmccart/p5.js/]] for creating graphic and interactive experiences, based on the core principles of Processing (being showcased at [[eyeo festival|http://eyeofestival.com/speaker/lauren-mccarthy/]] this week)
*Visit from SH Kim (Purdue U), prospective post-doc June 29
*GRAC update: PB to begin ~PhD w/ KM in Fall 2014, working on project with SC (U. Calgary)
*TM book updates?
*DIS Experience Night: Mon May 23 (Vancouver)
*VIS DC: awaiting results (no notification date set) (last year notification was ~1 month after submission)
*SIGGRAPH Vancouver Aug 10-14 (attending Exhibits?)
!References
<<bibliography>>
Meetings in Summer 2014:
*''4pm Wed June 18 @ Grindstone Caf駧
*Fri Jul 4 or Tue Jul 8?, Jul 22-25
1. Overview
*paper revisions (compare to r231 in SVN)
*cover letter (in SVN)
2. Viz Tasks for DR Data (fka DRITW)
*BELIV pre-paper talk slides v2 (see SVN link, sent via email)
*BELIV submission deadline June 30
*writing schedule, feedback from MS, SI
3. Science of Evaluation Workshop @ UCSD May 20-21
*ugh栐urdue doesn't recognize ~Canada-US Tax Treaty (reimbursement taxed at 30%, or $225/$750, possible to recoup taxes at year's end if I apply for ITIN from the IRS)
4. Empirical Work to Inform Visualization Design
*BELIV position paper with SC (Calgary), MT (U.Vic), BL (MSR)
5. Pulse (unchanged since last week)
*working on d3.js [[color stock tilemaps + (focus + context) + (linking + brushing)|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] (challenge: coordinated interactivity / highlighting in d3 with small multiples)
*week of Jun 2: met w/ UX designer, KT, dev team; some features coming online this month for new customers only (small multiple tilemaps and boxplots); technical product analyst is planning to conduct interviews, MB to join; tablet design considerations for energy managers conducting site inspections;
*[[original R/Shiny/ggplot implementation of portfolio visualization sandbox|http://spark.rstudio.com/mattbrehmer/ShinyPortfolio/]] (Feb 2014, Pulse-hosted instance no longer functioning due to R version / package version upgrades/conflicts made by Pulse Analytics team)
*CHI '15 paper on design? ~EuroVis '15 paper?
*TBD: 2nd internship as post-deployment analysis? timeline?
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager//
6. Task Typology
*Analyze as parent node name to consume-produce / other TM book updates
7. Misc
*urgent dental surgery afternoon Jun 19
*upcoming 0.5 MUX slot Jul 02: what to discuss?
*SH Kim (Purdue U) accepts post-doc position
*DIS Experience Night: Mon May 23 (Vancouver)
*VIS DC: awaiting results (no notification date set) (last year notification was ~1 month after submission)
*SIGGRAPH Vancouver Aug 10 - 14 (attending Exhibits?)
*upcoming travel: Oct ~10 - ~20 (not booked yet)
!References
<<bibliography>>
Meetings in Summer 2014:
*Jul 23 12:30 @UBC
1. Overview
*camera ready revisions in SVN / PCS (due Aug 1): removed mention student study; no room to mention of v5 / entity extraction
*video preview due Aug 1
2. VIS DC - Saturday Nov 8
*[[updated manuscript|http://people.cs.ubc.ca/~brehmer/proj/visdc.pdf]] due July 31: updated references, swapped order of interview study and energy management design study to reflect current chronology, added BELIV submission citation
*decision to include poster required by July 31
*video preview due Aug 1
3. [[BELIV 2014|http://beliv.cs.univie.ac.at/]] - Monday Nov 10
*reviews expected Aug 1
*(no review requests?)
4. Pulse
*last week: demo and think-aloud interviewed four members of Pulse's client services team re: new Energy Manager for Utilities (EMU) tool (limited feature set), with JN (UX designer) and JM (technical product analyst)
*I spoke at MUX last week about the project
**KB on connecting w/ [[UBC sustainability, architecture researchers|http://cirs.ubc.ca/research/research-area/visualization-tools-and-community-engagement#ParticipatoryFloodManagement]], design studies in CHI
*this week: debrief with JN, JM, CC (EMU engineering lead)
*following the next milestone: demos and think-aloud interviews with additional members of client services team, demos to external clients
*MB to demo / share notes with TM, get feedback
**juxtaposing statistical summaries with time-varying aggregates: boxplots and heatmaps
***previously, I had mocked up a d3.js [[color stock tilemaps + (focus + context) + (linking + brushing)|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] (challenge: coordinated interactivity / highlighting in d3 with small multiples)
**how can I justify "dual-axis w/ heterogeneous units for time series charts considered harmful, despite domain convention"?
***[[Hadley Wickham's take|http://stackoverflow.com/questions/3099219/how-to-use-ggplot2-make-plot-with-2-y-axes-one-y-axis-on-the-left-and-another]]
***<<cite Few2008a bibliography:Bibliography>> on [[sual-scaled charts|http://www.perceptualedge.com/articles/visual_business_intelligence/dual-scaled_axes.pdf]]
***<<cite Aigner2011b>> on [[emprically validating Bertin's indexing method|http://publik.tuwien.ac.at/files/PubDat_198120.pdf]]
***<<cite Isenberg2011a>> on [[dual axes, homogeneous units, superimposed focus and context|https://www.lri.fr/~isenberg/publications/papers/Isenberg_2011_ASO.pdf]]
*August: MB to start writing a manuscript summarizing design / findings to date
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager// - timing TBD
5. Misc
*CPSC 547 registration form for AC
*[[CANVAS|http://www.canvas2014.ca/index.php/schedule]] next week Jul 28 - Aug 1
*awaiting reimbursement for Science of Evaluation Workshop @ UCSD May 20-21
*registered for SIGGRAPH exhibits Vancouver Aug 12 - 14
*[[Tufte, Corum, Popova, Munroe|https://www.edwardtufte.com/tufte/course-register?course_id=1225®istration_type=student]] in Seattle for one-day course, Aug 7, $140 for students/faculty/postdocs
*upcoming deadlines: ~PacificVis 2015 (early Sept), CHI '15 (Sept 22), ~EuroVis '15 (early Dec)
*upcoming travel: Oct 6 - 20 (Ontario) Nov 6 - 20 (VIS, Europe, not booked yet)
!References
<<bibliography>>
Meetings in Summer 2014:
*Jul 23 12:30 @UBC
1. Overview
*camera ready revisions submitted
*video preview submitted
2. VIS DC - Saturday Nov 8
*[[updated manuscript|http://people.cs.ubc.ca/~brehmer/proj/visdc.pdf]] due July 31
*decision to include poster required by July 31
*video preview due Aug 1
3. [[BELIV 2014|http://beliv.cs.univie.ac.at/]] - Monday Nov 10
*reviews expected Aug 1
4. Pulse
*two weeks ago: demo and think-aloud interviewed four members of Pulse's client services team re: new Energy Manager for Utilities (EMU) tool (limited feature set), with JN (UX designer) and JM (technical product analyst)
*last week: debrief with JN, JM, CC (EMU engineering lead)
*following the next milestone: demos and think-aloud interviews with additional members of client services team, demos to external clients
*MB to demo / share notes with TM, get feedback
**juxtaposing statistical summaries with time-varying aggregates: boxplots and heatmaps
***previously, I had mocked up a d3.js [[color stock tilemaps + (focus + context) + (linking + brushing)|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] (challenge: coordinated interactivity / highlighting in d3 with small multiples)
**how can I justify "dual-axis w/ heterogeneous units for time series charts considered harmful, despite domain convention"?
***[[Hadley Wickham's take|http://stackoverflow.com/questions/3099219/how-to-use-ggplot2-make-plot-with-2-y-axes-one-y-axis-on-the-left-and-another]]
***<<cite Few2008a bibliography:Bibliography>> on [[sual-scaled charts|http://www.perceptualedge.com/articles/visual_business_intelligence/dual-scaled_axes.pdf]]
***<<cite Aigner2011b>> on [[emprically validating Bertin's indexing method|http://publik.tuwien.ac.at/files/PubDat_198120.pdf]]
***<<cite Isenberg2011a>> on [[dual axes, homogeneous units, superimposed focus and context|https://www.lri.fr/~isenberg/publications/papers/Isenberg_2011_ASO.pdf]]
*August: MB to start writing a manuscript summarizing design / findings to date
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager// - timing TBD
5. Misc
*[[CANVAS|http://www.canvas2014.ca/index.php/schedule]] Jul 28 - Aug 1
*reimbursed for Science of Evaluation Workshop @ UCSD May 20-21
*registered for SIGGRAPH exhibits Vancouver Aug 12 - 14
*registered for [[Tufte, Corum, Popova, Munroe|https://www.edwardtufte.com/tufte/course-register?course_id=1225®istration_type=student]] in Seattle for one-day course, Aug 7, $140 for students/faculty/postdocs
*upcoming deadlines: ~PacificVis 2015 (early Sept), CHI '15 (Sept 22), ~EuroVis '15 (early Dec)
*upcoming travel: Oct 6 - 20 (Ontario) Nov 6 - 20 (VIS, Europe, not booked yet)
!References
<<bibliography>>
Meetings in August 2014:
*Aug 21 4pm @ UBC
*next meeting TBA? Week of Sept 2?
1.BELIV 2014 - Monday Nov 10
*DR Vis tasks: revisions and cover letter due Sept. 1
**visualization idioms in Table 1
**number of dimensions / items: orders of magnitude vs. absolute values in Table 1
**//On Controlled Experiments & the ~DR-Naﶥ//
**Lewis et al's quality judgment as a low-level task or is are quality judgments orthogonal to task?
**word wrapping in cover letter: form question on submission site doesn't recognize newline characters, hence the "/ /" breaks
*~Pre-Design Empiricism position paper (w/ SC, Bl, MT) accepted
**to do: prepare talk / poster (format TBD)
2. Pulse
*following the next milestone: demos and think-aloud interviews with additional members of client services team, demos to external clients. ''update'': //implemention still won't load a __real__ (large) portfolio//
**juxtaposing statistical summaries with time-varying aggregates: boxplots and heatmaps. previously, I had mocked up a d3.js [[color stock tilemaps + (focus + context) + (linking + brushing)|http://www.cs.ubc.ca/~brehmer/research/d3portfolio/]] (challenge: coordinated interactivity / highlighting in d3 with small multiples). ''update'': //huge leap from introductory / basic d3.js (e.g. [[Scott Murray's textbook|http://chimera.labs.oreilly.com/books/1230000000345]] and [[dashingd3js tutorials|https://www.dashingd3js.com/table-of-contents]]) and [[reverse-engineering / appropriating mbostock's designs|http://bl.ocks.org/mbostock/4061502]]. I need to learn more, seek out intermediate design resources. I'm also learning [[processing.js|http://processingjs.org/]] (easier to learn, but smaller community)//
**how can I justify "dual-axis w/ heterogeneous units for time series charts considered harmful, despite domain convention"? ''update'': //normalization still not implemented.//
*--August: MB to start writing a manuscript summarizing design / findings to date--. ''update'': //still more design iteration to be done (on my part and Pulse's)//
*Pulse customer webinar (general energy utility / marketing audience), delayed to launch of redesigned //Energy Manager// - timing TBD
3. VIS '14
*[[IEEE VIS papers + video previews posted|http://ieeevis.org/year/2014/info/overview-amp-topics/accepted-papers]]
*to do: register, book flights, airbnb
*on what day is the VAD tutorial?
*interesting ~InfoVis papers:
**<<cite Huron2014a bibliography:Bibliography>> examining the visual mapping process using coloured wooden tiles (SC spoke of this at ~CANVAS2014), implications for [[Task Typology|Task Characterization: Meta Analysis]] postscript
**<<cite Kindlmann2014>>'s algebraic process for visualization design. Their FW is exciting and closely related to DR, tasks, DR technique + visualization idiom choices. Slight misinterpretation of our task typology paper?
**<<cite Saket2014a>> uses our task typology to characterize node, node-link, and node-link-group visualization tasks
**<<cite Mckenna2014>>'s design activity framework (DSM + NM); [[supplemental materials containing table of 100 methods and definitions|http://mckennapsean.com/projects/design-activity-framework/suppl-mat.pdf]]
**<<cite Sedlmair2014>> on tasks for parameter space analysis (and their commentary on our task typology)
*VIS tutorials:
**//Everything Except the Chart: Developing Complex Web-based Visualizations// - D. Baur, M. Stefaner
**//"Where Do I Start?" Practical Methods for the Design Studies in Information Visualization and Visual Analysis// - L. ~McNamara, K. Cole, S. ~Stevens-Adams
4. Overview
*to do: prepare talk - practice talk MUX Oct 29
5. VIS DC - Saturday Nov 8
*to do: prepare final version of poster, prepare presentation
6. Misc
*notes from [[See, Think, Design, Produce]] - Aug 7, Seattle
**met w/ JB, UBC Stats
*[[keyvis.org|http://www.keyvis.org/]]
*[[performance vis|http://idav.ucdavis.edu/~ki/STAR/]], [[financevis.net|http://financevis.net/]], [[seealso.org|http://seealso.org/]] (wikipedia vis)
*upcoming travel: Oct 6 - 20 (Ontario) Nov 6 - 21, Europe, not booked yet)
*upcoming deadlines: ~PacificVis 2015 (early Sept), CHI '15 (Sept 22), ~EuroVis '15 (early Dec)
*Shneiderman visit / ~InfoVis group gathering ~mid-Sept
!!References
<<bibliography>>
Meetings in September 2014:
*Sept 5 2:30pm @ UBC
*regular meeting time TBD
1. CPSC 547 geust lecture on resources / tools / packages
*[[Visualization Design Resources]]
2. Pulse EMU
*meeting w/ JN (UX designer) and JC (client services) next Tuesday (Sept 9) in a feedback session on EMU design; 1:1 w/ JN to discuss boxplots
*Recently: working on the design space of interactive boxplots, boxplots as supplemental visualizations:
**[[Simple Box Plot w/ Point Hover|http://bl.ocks.org/mattbrehmer/12ea86353bc807df2187]]
**[[Box Plot w/ Brushing|http://bl.ocks.org/mattbrehmer/8be29724bdd7a63ff41d]]
**[[Color Stock Chart (Small Multiples)|http://bl.ocks.org/mattbrehmer/82cf72481d4753eca1cf]]
**[[Juxtaposed Focus + Context Box Plots|http://bl.ocks.org/mattbrehmer/287e44c9a12151967874]]
3. BELIV 2014 - Monday Nov 10
*final notification Sept 11
*DR Vis Tasks conditionally accepted, PDE position paper accepted
*to do: prepare talks / poster (format TBD)
4. VIS '14
*to do: book flights, airbnb
*recently read <<cite Sacha2014 bibliography:Bibliography>> on the //Knowledge Generation Model for Visual Analytics//: [[notes|Task Characterization: Meta Analysis]]
5. Overview
*to do: prepare talk - practice talk date/time TBD (bumped from MUX Oct 29 for CPSC 544 session)
*borrowing from TM's HOPE X slides / last year's CPSC 344 slides
*to read related work appearing @ VIS 14: <<cite Cui2014>>, <<cite Alexander2014>>
5. VIS DC - Saturday Nov 8
*to do: prepare final version of poster, prepare presentation; practice talk date/time TBD
*borrowing from thesis proposal slides, focusing on Pulse EMU project
7. Job Search
*research internship summer 2015?
*industry R&D job search in Vancouver, Seattle, PNW for early 2016 start
6. Misc
*~InfoVis group retreat? Shneiderman visit?
*upcoming travel: Oct 6 - 20 (Ontario) Nov 6 - 21, Europe, not booked yet)
!!References
<<bibliography>>
Meetings in September 2014:
*Sept 5 2:30pm, ''Sept 12 4pm'', next meeting TBD
*regular meeting time TBD after VIS?
1. CPSC 547 guest lecture on resources / tools / packages
*[[InfoVis Group Visualization Design Resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]] (also updated link CSS on group page)
*to speak about D3.js, Processing(.js,P5.js), R/Shiny, Tableau
2. Pulse EMU
*met w/ JN (UX designer) and JC (client services) Tuesday in a feedback session on EMU design; 1:1 w/ JN to discuss [[interactive boxplots|http://bl.ocks.org/mattbrehmer/]]
*outlier emphasis, superimpose outlier glyphs on heatmap (depending on direction of outlier)
3. BELIV 2014 - Monday Nov 10
*[[DR Vis Tasks paper page|http://www.cs.ubc.ca/labs/imager/tr/2014/DRVisTasks/]], [[PDE position paper paper page|http://www.cs.ubc.ca/labs/imager/tr/2014/PDE/]], updated group website
*to do: prepare talks / poster (format TBD) - DR Vis Tasks practice talk Fri Oct 24 10:30am
4. VIS '14
*flights booked (Lufthansa/Air Canada, Nov 5 to Paris, Nov 24 from D쳳eldorf) - possible VIS DC reimbursement of $100 for travel (if they accept Star Alliance codes shares, organizers specified that we must use a US carrier, but flight itineraries/costs didn't work out)
*to do: book airbnb (VIS DC to cover $644)
*VIS DC to cover $300 / $400 registration (student speaker rate)
5. VIS DC - Saturday Nov 8
*to do: prepare final version of poster
*practice talk date/time Thu Sept 18 3:30pm: [[review of slides|https://dl.dropboxusercontent.com/u/6397998/14.11.08-mb-visdc.pdf]] (borrowing from thesis proposal slides, focusing on Pulse EMU project)
*FF video due Oct 7
6. Overview
*to do: prepare talk - practice talk Fri Oct 3 10:30am (to do: email JS, SI to verify that time slot works)
**borrowing from TM's HOPE X slides / last year's CPSC 344 slides
*FF video due Oct 7
*to read related work appearing @ VIS 14: <<cite Cui2014 bibliography:Bibliography>>, <<cite Alexander2014>>
7. Job Search
*research internship summer 2015?
*industry R&D job search in Vancouver, Seattle, PNW for early 2016 start
8. Misc
*~InfoVis group retreat details? FW / collboration ideas for projects w/ JF, MB, HL
*upcoming travel: Oct 6 - 20 (Ontario), Nov 6 - 24 (Europe)
!!References
<<bibliography>>
Meetings in September 2014:
*Sept 5 2:30pm, Sept 12 4pm, ''Sept 26 2:30''
1. Overview
*in "text / documents / images" session, Fri Nov 14 8:30am (Overview talk is first in session)
*to do: review slides / notes
**borrowing from TM's HOPE X slides / last year's CPSC 344 slides, pre-paper slides
*practice talk Fri Oct 3 10:30am, SI can attend via Skype, JS cannot
*practice talk #2 in MUX, Wed Oct 22 2pm (JS can attend this time)
*FF video submitted
*to read related work appearing @ VIS 14: <<cite Cui2014 bibliography:Bibliography>>, <<cite Alexander2014>>
2. BELIV 2014 - Monday Nov 10
*to do: prepare talks / poster (format TBD) - DR Vis Tasks practice talk Fri Oct 31 10:30am
3. VIS DC - Saturday Nov 8
*to do: revise slides / script based on retreat feedback, schedule another practice talk?
*to do: prepare final version of poster (Q: how long in advance is needed for printing?)
*no FF needed
4. Pulse EMU
*[[feedback from meeting w/ JC (Pulse client services)|Pulse-JC-14.09.09]]
*[[interactive boxplots|http://bl.ocks.org/mattbrehmer/]]
*outlier emphasis, superimpose outlier glyphs on heatmap (depending on direction of outlier)
*annotation and alerts, propagating these from single building, short time granularity to aggregate views
5. Job Search
*research internship summer 2015? MSR: emailing BL / NHR
*industry R&D job search in Vancouver, Seattle, PNW for early 2016 start
6. Misc
*feedback on vis resources talk / website?
*[[Task typology used in molecule parametrization vis|http://dare.uva.nl/cgi/arno/show.cgi?fid=529854]]
*BREB ethics renewal - Oct 31
*funding: switch from NSERC ~PGS-D to 4YF in January
*upcoming travel: Oct 6 - 20 (Ontario), Nov 6 - 24 (Europe)
*Here's my more detailed schedule with suggested meeting times:
**Fri Sept 26 (today): meeting at UBC 2:30 - 3:30
**Fri Oct 3: meet at UBC in afternoon TBD (debrief from AM Overview practice talk)
**Mon Oct 6: (flying to Ontario), working remotely Oct 7-10
**Fri Oct 10: meet via Skype?
**Mon Oct 13: thanksgiving holiday, visiting family Oct 14-17, working while traveling via train
**Mon Oct 20: fly back to YVR
**Wed Oct 22: Overview practice talk #2 in MUX + JS via Skype, debrief after?
**Mon Oct 27: meet 3:30 @ UBC to prepare DRITW talk?
**Fri Oct 31: DRITW BELIV practice talk in group meeting, debrief after?
**Mon Nov 3: meet 3:30 @ UBC
**Wed Nov 5: flying to CDG, VIS Nov 8 - 14, Vacation / visiting family Nov 15-23
**Mon Nov 24: return flight from DUS
**Mon Dec 1: meet 3:30 at UBC
!!References
<<bibliography>>
Meetings in October 2014:
*''Oct 10'', Oct 22 (post-MUX debrief), Oct 27 (TBD) or Oct 31 (TBD)
1. Overview
*in "text / documents / images" session, Fri Nov 14 8:30am (Overview talk is first in session)
*to do: revise slides based on JS feedback (10/09):
>''JS''://It's a bit of a slow start. Might be worth showing final interface or mentioning Pulitzer finalist right at beginning to motivate//
>
>//Discussion of tree redesign mentions exhaustive review. Worth emphasizing, I think, as this is common, unexpected, and I suspect counter-intuitive to visualization designers//
>
>//Could trim acknowledgements/timeline to one slide and a couple sentences to save time and improve pacing.//
*to read related work appearing @ VIS 14: <<cite Cui2014 bibliography:Bibliography>>, <<cite Alexander2014>>
2. BELIV 2014 - Monday Nov 10
*tentative format: 5 min talk + panel discussion
*BELIV practice talks x2 (DR Vis Tasks / PDE) - Oct 22 2pm in MUX
*BELIV DR Vis Tasks slides + speaker notes in dropbox link (see email)
*BELIV DR Vis Tasks: fixed embedded fonts issue
*BELIV PDE: MB to give practice talk to co-authors via Skype (date/time TBD draft slides / speaker notes done)
3. VIS DC - Saturday Nov 8
*presentation revised, re-recorded, (mp4 in dropbox, see email)
*to do: schedule another practice talk?
*to do: prepare final version of poster
4. Pulse EMU
*Pulse devs have added interactive tooltips to boxplots
*my [[interactive boxplot sketches|http://bl.ocks.org/mattbrehmer/]] on bl.ocks
*annotations and alerts, propagating these from single building, short time granularity to aggregate views (ongoing discussion)
5. Job Search
*email exchange w/ NHR on 10/07: will be at VIS, will meet to discuss upcoming projects including domain collaborations in neuroscience and data journalism
*industry R&D job search in Vancouver, Seattle, PNW for early 2016 start
6. Misc
*funding: switch from NSERC ~PGS-D to 4YF in January
*MB travel schedule: Oct 6 - 20 (Ontario), Nov 6 - 24 (Europe)
**Fri Oct 10: meet via Skype 2:30 Pacific
**Mon Oct 13: thanksgiving holiday, visiting family Oct 14-17, working while traveling via train Oct 14
**Mon Oct 20: fly back to YVR
**Wed Oct 22: BELIV practice talks in MUX, debrief after?
**Mon Oct 27: TBD
**Fri Oct 31: TBD
**Mon Nov 3: TBD
**Wed Nov 5: flying to CDG, VIS Nov 8 - 14, Vacation / visiting family Nov 15-23
**Mon Nov 24: return flight from DUS
**Mon Dec 1: meet 3:30 at UBC
!!References
<<bibliography>>
Meetings in November 2014:
*''Nov 3'', Tue Nov 25 㠆ri Nov 28?
1. Overview
*in "text / documents / images" session, Fri Nov 14 8:30am (Overview talk is first in session)
*ongoing talk rehearsal (17min)
*re: Gibson quote
*to read related work appearing @ VIS 14: <<cite Cui2014 bibliography:Bibliography>>, <<cite Alexander2014>>
2. BELIV 2014 - Monday Nov 10
*format: 5 min talk + panel discussion; ongoing talk rehearsal
*BELIV practice talks x2 (DR Vis Tasks / PDE) - Oct 22 2pm in MUX
*read papers appearing in same BELIV sessions: session on "Rethinking Evaluation Level: Abstracted Task vs In Situ Evaluation": <<cite Rind2014>>, <<cite Winters2014>>, <<cite Correll2014>>; session on "Experience Reports": <<cite Kim2014>>, <<cite Scholtz2014>>, <<cite Kaastra2014>>
*to read: <<cite Aupetit2014>> paper on DR
*to do: [[BELIV discussion points|https://docs.google.com/document/d/1LQUNYcQ7bgPrxI_0MeMDSt-3QQSO1Vwepj-1a_E1gH4/edit?usp=sharing]]
3. VIS DC - Saturday Nov 8
*poster printed (80$ from SML Buisness Ltd, Broadway and Cypress), borrowing poster tube from MUX; recieved feedback on poster from Joanna's LUNCH group
*[[VIS DC panel, schedule|https://docs.google.com/document/d/1J9yB6lMH1QgKWUYXB8UF83jti_8meF2clg_dwQepODA/edit]] announced
*ongoing talk rehearsal (17 min)
4. Pulse EMU
*update: external feedback interviews forthcoming (as of 10/31)
5. Job Search
*email exchange w/ NHR on 10/07: will be at VIS, will meet to discuss upcoming projects including domain collaborations in neuroscience and data journalism
*[[Tableau Research internship|https://ch.tbe.taleo.net/CH11/ats/careers/requisition.jsp?org=TABLEAU&cws=1&rid=3187]] posted
*[[Autodesk internship|http://www.autodeskresearch.com/collaborations/researchinternshipform]] posted
*industry R&D job search in Vancouver, Seattle, PNW for early 2016 start
*updated [[CV|http://cs.ubc.ca/~brehmer/mb_cv.pdf]], created [[resume|http://cs.ubc.ca/~brehmer/mb_resume.pdf]] (ࠃV), [[website|http://cs.ubc.ca/~brehmer/]]
6. Misc
*[[VIS blind lunch|http://vacommunity.org/IEEE+VIS+%2714+Blind+Lunch]] Tue, Nov. 11, w/ J. Dykes
*[[VIS 2014 Schedule Browser|http://gramaz.io/vis/2014-schedule.html]] by [[ Connor Gramazio|http://gramaz.io/]], CS ~PhD Candidate @ Brown
*spoke to MB this afternoon re: QDA
*ORS RISE renewal confirmed
*writing a CHI review (due Nov. 10)
*[[R Graph Catalog|http://shinyapps.stat.ubc.ca/r-graph-catalog/#]] - Shiny App, complement to //Creating More Effective Graphs// by Naomi Robbins; gallery / code maintained by Joanna Zhao and Jennifer Bryan (added to resources page)
*[[A Big Article About Wee Things|http://www.propublica.org/nerds/item/a-big-article-about-wee-things?utm_campaign=bt_twitter&utm_source=twitter&utm_medium=social]] by Lena Groeger (~ProPublica), article on small multiples, affordances
*funding: switch from NSERC ~PGS-D to 4YF in January
*MB travel schedule: Nov 6 - 24 (Europe)
!!References
<<bibliography>>
Meetings in December 2014:
*''Dec 1'', Dec 8, Dec 15
1. VIS - a more specific debrief
*''administrivia'': reimbusement requests submitted to UBC and UMBC (NSF DC funding)
*''VIS DC'': [[feedback from panelists|VIS DC Feedback]] (audio transcript)]
**ideas for Task Typology addendum (dissertation sub-chapter)
**ideas for Typology validation paper with CPSC 547 Fall 2015 course projects
**ideas for Pulse EMU 2015 ~InfoVis paper
**ideas for job search
*''BELIV''
>//"There was also a panel on tasks, which largely talked about task taxonomies. There was an odd lack of self-awareness on that panel, because for all the talk about tasks, there didn鴠seem to be much thought about what people would actually do with those taxonomies. Who are the users of the taxonomies? What are their tasks? Is any of this work actually useful, or is it just done for its own sake? That struck me as particularly odd as part of this event."// - [[(guess who)|https://eagereyes.org/blog/2014/vis-2014-monday]]
**journal version of DR Vis Tasks paper? TBA
*''Overview:'' feedback from JSY (Purdue)?, RH (Twitter)
*''MSR internship'' w/ BL, NHR; tentatively July - Sep 2015 with pre-design research occurring before; journalist users + visualization for presentation; potential follow-on from previous collaboration w/ BCK (Konstanz / formerly Purdue), ongoing collaboration w/ FA (Manitoba)
*lunches w/ MD (Potsdam), JD (City U London), CW (Oklahoma)
*chatted w/ LC, [[OpenVisConf|http://openvisconf.com/]] abstract submitted Nov. 17 / accepted talks TBA; abstract:
>//"The Why, What, and How of Visualization Analysis and Design
WHY do people use visualization? WHAT information does visualization communicate to its viewers, or WHAT new information can be produced via interaction with visualization? And HOW should visualization support a diverse set of tasks?
In this talk I will introduce a way to think about visualization analysis and design that revolves around a visualization user's tasks, asking Why, What, and How. I will do this by presenting two case studies: two projects from my doctoral work at the University of British Columbia, where I am a member of the ~InfoVis and Multi-modal User Experience research groups. Both case studies involve interactive web-based visualization tools intended to support exploratory and confirmatory data analysis by subject matter experts.
The first case study is about Overview (overviewproject.org), a visualization tool for investigative journalists, a tool that allows journalists to examine large text document collections emanating from leaks or Freedom of Information Act requests in a systematic way. Overview has helped journalists write a number of published stories, including a recent finalist for the 2014 Pulitzer Prize in journalism. However, it took four years and several iterations of design and analysis of Overview's adoption and use by journalists to fully realize which design decisions were effective, and which were not. In asking Why, What, and How, it was possible to glean several generalizable visualization design lessons.
The second case study will focus on the visualization design process, this time in the context of large-scale commercial energy consumption analysis. Working in collaboration with Pulse Energy (pulseenergy.com), I began by asking Why and What, studying the workflows of energy analysts at utility companies and other large organizations. Next, I asked How, rapidly generating and iterating on a set of "data sketches". I'll trace how these sketches evolved into interactive visualization prototypes and more recently into components of a web-based energy consumption analysis tool.
The answers to the questions of Why, What, and How form a language of visualization design that allows you to borrow ideas for your visualization from those designed for entirely different application domains."//
2. Pulse EMU
*update: first external feedback interview w/ SSD energy manager Tue Dec 9, 10am
*recent updates to EMU interface (brief demo)
* [[feedback from VIS DCVIS DC Feedback|VIS DC Feedback]] on what to include in this paper; pre-paper talk in Jan / Feb, ~InfoVis 14 submission
3. Job / Internship Search
*MSR (see above)
*[[Tableau Research internship|https://ch.tbe.taleo.net/CH11/ats/careers/requisition.jsp?org=TABLEAU&cws=1&rid=3187]] posted
*[[Autodesk internship|http://www.autodeskresearch.com/collaborations/researchinternshipform]] posted
*Adobe internship list sent out to IMAGER
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
4. Misc
*more active role in JF journalism project meetings?
*spoke to MB (UBC) Nov 3 re: QDA / work domain analysis for brain surgeons
*funding: switch from NSERC ~PGS-D to 4YF in January
*GRAC committee 2014-15, meetings begin this month, application reviews begin January
Meetings in December 2014:
*Dec 1, ''Dec 15'' 2pm
1. Pulse EMU
*[[Pulse acquired by EnerNOC|http://www.pulseenergy.com/pulse/a-letter-from-our-co-founder-and-ceo-david-helliwell/]] Dec 2
*EMU demos: external feedback interview w/ SSD energy manager Tue Dec 9; demo to facilities manager at Loughheed Town Centre (Burnaby) Dec 9
*[[feedback from VIS DCVIS DC Feedback|VIS DC Feedback]] on what to include in this paper; pre-paper talk week of Jan 23, ~InfoVis 14 submission
*ranking visualization:
**began preparing datasets for interactive ranking vis based on earlier R/Shiny prototype (Pulse normalized energy consumption, [[QS university rankings|http://www.topuniversities.com/qs-world-university-rankings]], [[music review rankings|http://www.albumoftheyear.org/ratings/overall/2014/]])
**check out [[lineup.js|http://caleydo.github.io/lineup.js/demo/#]] by JKU's SG (weighted multi-attribute ranks, but no bump charts)
2. Job / Internship Search
*MSR: w/ BL, NHR; tentatively July - Sep 2015 with pre-design research occurring before; journalist users + visualization for presentation; potential follow-on from previous collaboration on [[VisJockey|http://bib.dbvis.de/uploadedFiles/VisJockey_C_J2014.pdf]] w/ BCK (Konstanz / formerly Purdue), ongoing collaboration w/ FA (Manitoba)
*chatted about careers w/ JM Dec 02
*Pulse's KT inquired about another internship, open post-grad job offer
*Dec. 12 email from RB, Oculus Info, Toronto re: part-time remote work
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
*read [[LC's advice re: vis consulting|http://blogger.ghostweather.com/2013/11/data-vis-consulting-advice-for-newbies.html]]
3. VIS 2014
*reimbusement requests submitted to UBC and UMBC (NSF DC funding), awaiting reimbursement
*VIS DC: [[feedback from panelists|VIS DC Feedback]] (audio transcript)] - thoughts?
4. Misc
*[[OpenVisConf|http://openvisconf.com/]] abstract submitted (rejected); J. Heer and S. Ortiz to present keynote talks
*met w/ JG + AC (BCCDC), PB (UBC) Dec 12 re: vis project at the CDC
*CPSC 547 project presentations
*UMD's MA Yal穮 (UMD) and task typology interactive survey site (see [[alternative tree vis survey|http://keshif.me/demo/treevis.html]] and [[Keshif|http://keshif.me/]])
*JF + TM timeline vis project ([[VisJockey|http://bib.dbvis.de/uploadedFiles/VisJockey_C_J2014.pdf]] may also be relevant RW here)
*MB work domain analysis for brain surgeons (delayed)
*~EuroVis review requests x 2
*[[tutorial slides|http://goo.gl/fOJP7j]] from LM (Sandia)'s IEEE VIS tutorial on work domain analysis and cognitive task analysis in design studies
*learning [[p5.js|http://p5js.org/reference/#/p5/set]]: [[Processing scatterplot maps translated into p5.js|http://bl.ocks.org/mattbrehmer/a129bf7a88f83ef1d0d5]]
*tried out [[bertifier.com|http://bertifier.com/]], made [[whistler monthly snowfall tabular vis|https://twitter.com/mattbrehmer/status/540972356294815744]]
*[[InfoVis group resources page|http://www.cs.ubc.ca/group/infovis/resources.shtml]] updated
*GRAC committee 2014-15, meetings begin this month, application reviews begin January: one secondary review request sent to TM 12/15
*funding: switch from NSERC ~PGS-D to 4YF in January
Meetings in January 2015:
*''Fri Jan 9'' 3:10pm, Mon Jan 19 2:30
1. Pulse EMU
*pre-paper talk (Jan 23) [[draft slides|https://dl.dropboxusercontent.com/u/6397998/15.01.09-pre-paper-talk-draft.pdf]] - dense as Plutonium? a 2 hour pre-paper talk? (to discuss at length during out meeting on Jan 19)
**Pulse's KT, JN (EMU UI designer), CC (lead EMU dev) to join via Skype (?)
*relative + absolute ranking visualizations created over the holidays, shared with Pulse:
**[[SoundConsensus|http://mattbrehmer.github.io/SoundConsensus/]]
**[[University Rankings|http://people.cs.ubc.ca/~brehmer/demo/universities/]]
2.Job / Internship Search
*MSR - meeting next week via Skype w/ BL, NR (internship topic TBD, likely no longer about journalism, but still about presentation / storytelling)
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
3.GRAC
*meetings Monday 1-2
*8 of 15 HCI / Vis applicants reviewed, some secondary requests sent to TMM
4.JF Timelines Vis
*met w/ PR, UBC SCARP ~PhD student 15.01.08 re: global social housing timelines, needs a non-programming tool to develop vis for presentation (TBD w/ JF)
5. Misc
*funding: switch from NSERC ~PGS-D to 4YF in January
*I won a spot in the CHI SV lottery (one of 136, 680 on waitlist) - Apr 18-23 in Seoul
*VIS expense reimbusement request received by UMBC (NSF DC funding), awaiting reimbursement; other half of VIS expenses already reimbursed by UBC
*met w/ JG + AC (BCCDC), PB (UBC) Dec 12 re: vis project at the CDC
*UMD's MA Yal穮 (UMD) and [[task typology interactive survey site|http://keshif.me/demo/vis_tasks.html]]
*M Bork. work domain analysis for brain surgeons (delayed)
*~EuroVis review requests x 3 (due Jan 19)
Meetings in January 2015:
*Fri Jan 9 3:10pm, ''Mon Jan 19 2:30''
1. Pulse EMU
*pre-paper talk (take 2): [[slides|https://dl.dropboxusercontent.com/u/6397998/15.01.19-pre-paper-talk-draft.pdf]]
*group pre-paper talk (Jan 23)
*pre-paper meeting 3pm Thu Feb 5 @ Pulse w/ KT, JN (EMU UI designer), CC (lead EMU dev)
2.Job / Internship Search
*MSR - meeting tomorrow 10am via Skype w/ BL, NR
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
*email from HP (Harvard) re: postdoc
*email from EB (NYU Poly) re: postdoc
3.GRAC
*meeting Monday Jan 26 1-2
*19 of 23 HCI / Vis applicants reviewed (2 offers made), 8 review requests sent to TMM
4.JF Timelines Vis
*met w/ PR (UBC SCARP) and JF last Thursday, demoed timeline vis tool (TBD during Wednesday meeting)
5. Misc
*funding: switch from NSERC ~PGS-D to 4YF + supervisor topup in January
*I won a spot in the CHI SV lottery - Apr 18-23 in Seoul - tentatively traveling Apr 8 - 26
*VIS expense reimbusement request received by UMBC (NSF DC funding), awaiting reimbursement (email update from UMBC sent last week)
*met w/ M Bork last week re: task analysis for Glue paper
*~EuroVis reviews x3 complete (due Jan 19)
Meetings in February 2015:
*''Mon Feb 16 2:30pm'', ''Mon Feb 23 2:30''
1. Pulse EMU
*[[currently writing|https://www.sharelatex.com/project/54d55c089b8eb8a228028597]] / paper-ifying pre-paper slides
*draft 1 by 15.02.23? TM pass? then JN pass?
*draft 2 by 15.03.10?
*~InfoVis group read 15.03.13
*draft 3, KT pass following week?
*abstract deadline 15.03.21 5pm
*draft 4, JN, TM passes the following week?
*submission deadline 15.03.31 5pm
*JD volunteered to read at some point in March
2.Job Search
*MSR internship Jun 6 - Sept 4 (13 weeks); last week: preparing immigration and onboarding documents; research topic to relate to visual encodings and interactions w/ timelines
*MSR - TM to visit
*Tableau - TM to visit / speak to LW et al.
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
*15.01.30: attended 'Research at Google' seminar, followed up via email w/ recruiter
*15.02.13: spoke w/ recruiter at FCV (Canadian digital design firm)
*HP (Harvard) re: postdoc - follow-up?
*EB (NYU Poly) re: postdoc
**15.02.09: tentatively going to give a remote seminar in EB's Vis class re: Overview (at some point?)
3.GRAC
*offers, vis prospects, visit day, co-supervision
4. Misc
*to do: TLC project page (tomorrow)
*to do: infovis group website software, paper pages
*funding: switched from NSERC ~PGS-D to 4YF + supervisor topup in January (payroll change forms indicate Apr for supervisor topup, Aug for 4YF - should both be Dec 2015)
*CHI 2015 - Apr 18-23 in Seoul - traveling in Japan / Korea Apr 6 - 26
*VIS 14 expenses: awaiting VIS DC reimbursement from UMBC / NSF
Meetings in February 2015:
*Mon Feb 16, ''Mon Feb 23 2:30pm''
1. Pulse EMU
*[[first draft complete|https://www.sharelatex.com/project/54d55c089b8eb8a228028597]] / edit token to TM
*JN pass later this week
*draft 2 by 15.03.10?
*~InfoVis group read 15.03.13
*draft 3, KT pass following week?
*abstract deadline 15.03.21 5pm
*draft 4, JN, TM passes the following week?
*submission deadline 15.03.31 5pm
*JD volunteered to read at some point in March
*external readers?
2.Job Search
*MSR internship Jun 6 - Sept 4 (13 weeks)
*MSR - TM visit debrief?
*Tableau - TM visit debreif?
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
3.GRAC
*offers, vis prospects, visit day, co-supervision
4. Misc
*funding: switched from NSERC ~PGS-D to 4YF + supervisor topup in January (payroll change forms indicate Apr for supervisor topup, Aug for 4YF - should both be Dec 2015)
*CHI 2015 - Apr 18-23 in Seoul - traveling in Japan / Korea Apr 6 - 26
*VIS 14 expenses: awaiting VIS DC reimbursement from UMBC / NSF
Meetings in March 2015:
*''Mon Mar 9 2:30pm'', Mon Mar 16, Mon Mar 23, Mon Mar 30
1. Pulse EMU
*[[JN 1st commenting pass complete|https://www.sharelatex.com/project/54d55c089b8eb8a228028597]] / edit token to MB, send to group + Pulse's CC this Wednesday
*~InfoVis group read Friday (15.03.13)
*awaiting larger anonymized portfolio data set in EMU from Pulse's CC
*awaiting information from KT, AL re: # of large portfolios, # of portfolio energy workers
*draft 3, KT pass next week?
*abstract deadline 15.03.21 5pm (already submitted, email verification for KT, JN needed)
*draft 4, JN, TM passes the following week?
*submission deadline 15.03.31 5pm
*JD volunteered to read at some point in March
*external readers? JD? JM? others?
2. TLC
*[[paper draft|https://www.sharelatex.com/project/54ee4b9bc0e23207062cea10]] status
*benchmark evaluation status
*meeting last week w/ PS (The Tyee magazine)
*usage by humanities researcher in the UK to create timeline of British deaf church
*shout-out by N. Diakopolous at NICAR
3. Job Search
*MSR internship Jun 6 - Sept 4 (13 weeks)
*currently finalizing J1 visa documents
*industry R&D job search in Vancouver, Seattle, PNW for 2016 start
*received follow-up from RB (Uncharted / Oculus Info)
*to do: chat w/ HP (Harvard)
4. GRAC
*offers, visit day debrief
*Skype call w/ ZL last Monday, introduced to SI, WD
*met last week w/ PL (local prospective ~InfoVis grad 2016-17)
5. Misc
*2 IVJ BELIV special issue review requests due Apr 3
**DRITW / DR Vis Tasks Journal version?
*N. Diakopolous at CSCW
*meeting w/ B. Bailey Thursday
*funding: switched from NSERC ~PGS-D to 4YF + supervisor topup in January (payroll change forms indicate Apr for supervisor topup, Aug for 4YF - should both be Dec 2015)
**to do: MB speak w/ JP re: MSR internship + pay schedule
*CHI 2015 - Apr 18-23 in Seoul - traveling in Japan / Korea Apr 6 - 26
*VIS 14 expenses: VIS DC reimbursement from UMBC / NSF finally arrived last week!
Meetings in March 2015:
*Mon Mar 9, Fri Mar 20, ''Mon Mar 30 2:30pm'', next meeting week of Apr 27
1. Pulse EMU
*manuscript revised, MB to do final proofread
*revised [[d3 coordinated auxiliary boxplot design|http://bl.ocks.org/mattbrehmer/287e44c9a12151967874]]
*supplemental video edits? re-record audio? disclaimer about visenc? anonymized portfolio mention?
*JN question re: sharing manuscript w/ clients
2. TLC
*submitted
*~CloudLines RW?
3. MSR meeting
*Thursday via Skype 10-12:30
*send TLC manuscript to BL, NHR in advance of meeting? refer to Overview paper?
*agenda:
>//What should be on the agenda for the meeting? After visiting MSR in February, Tamara mentioned a potential for collaborations with us beyond my internship project, such as possible follow-on work relating to ~TimeLineCurator and Overview. With respect to the internship project, I expect that we would want to address (1) the current state of the art in visualization authoring / storytelling tools, (2) the design space of timeline visual encodings and interactions, (3) target user community, and (4) Tamara's involvement during / after the internship.//
4. Misc
*[[YVR Hacks/Hackers meetup Apr 16|http://www.meetup.com/HacksHackersVancouver/events/221489375/]] (at CHI, but JF + TM could attend)
*2 IVJ BELIV special issue review requests due Apr 3
*4YF funding suspension request submitted: Jun - Aug
*CHI 2015 - Apr 18-23 in Seoul - traveling in Japan / Korea Apr 6 - 26
*committee meeting 2:30 Thu Apr 7
Meetings in May 2015:
*''Fri May 1'', Wed May 6 (MSR), Mon May 11, Thu May 21, Wed May 27,
1. timeline internship project MSR
*debrief from CHI meetings w/ BL, NHR, BB, & TD: sketches and slides
*upcoming meeting: Wednesday, May 6 (at MSR or UBC? TBD)
2. Reviewing
*2 IVJ BELIV special issue reviews completed Apr 3
*3 ~InfoVis reviews due May 8
**Tufte / ~Graves-Morris - VDQI Vol 2 citation?
*2 VAST reviews due May 9
3. Misc
*my [[CHI Notes|CHI2015]], [[R. Kosara's CHI blog post|https://eagereyes.org/blog/2015/conference-report-chi-2015]]
*[[OpenVisConf videos posted|http://openvisconf.com/]]
*consensus ranking for multiple rankings - ~InfoVis poster? (June 26 deadline)
*Vis for tracking student progress in online education - meeting May 22 (via J Gardy)
*4YF funding suspended: Jun - Aug
*committee meeting 3:30 Thu May 7; annual progress report submitted Apr 28
*upcoming travel: New Orleans May 29 - Jun 2
Meetings in May 2015:
*Fri May 1, Wed May 6 (MSR), Thu May 7 (committee meeting), ''Thu May 21'', Wed May 27,
1. timeline project
*(see also email that I sent on Friday)
*lots of notes in Google Drive folder
*lit review of "timelines for analysis" papers: designs and "insights/findings"
**Continuum, Similan, Lifelines2
*further inspiration from //Cartographies of Time//: Condorcet's 3-dimensional timelines, the ~Polish-American System of Chronology, pictographic timelines w/ varying duration units used by 18th century Oregonian missionaries
*dataset search
**role of TLC for dataset generation, [[Wikipedia timeline list|http://en.wikipedia.org/wiki/List_of_timelines]] scraping
**~TimeFlow timelines
**"composers + their works" dataset: spans and events (used in Continuum / Andre et al, UIST 2009)
2. Committee Meeting Debrief
*annual report form submitted to Joyce / Grad studies?
*thesis defense timeline / last day to submit graduation applications is usually 3rd week of April
**re-assess pending June 6 VIS decision
*follow-up on job search discussion
3. Misc
*[[spoke at VDV meetup|http://people.cs.ubc.ca/~brehmer/pubs/15.05.19-VDV.pdf]] May 19 (debrief?)
*TM cali debrief
*Skype call last week w/ M. Borkin re: qualitative methods, shared final ~InfoVis submission + supplemental materials
*consensus ranking for multiple rankings - submit as ~InfoVis poster if Pulse paper rejected (June 26 deadline)
*watched most (17 / 21) of the [[OpenVisConf videos|http://openvisconf.com/]]
*Vis for tracking student progress in online education - meeting May 22 (via J Gardy)
*4YF funding suspended: Jun - Aug
*upcoming travel: New Orleans May 29 - Jun 2
*MSR internship June 8 - Sept 4
Meetings in May 2015:
*Fri May 1, Wed May 6 (MSR), Thu May 7 (committee meeting), Thu May 21, ''Wed May 27''
1. timeline project
*lots of notes in Google Drive folder
**Fisheye F+C timelines, ongoing thread w/ BB
***how can absolute a priori importance of [an] event(s) be determined upon load / upon interaction?
***how to show an entire timeline on one screen? [or], what are the best off-screen affordances / information scent indicators?
***can the fisheye notion be extended to types of events rather than single events? how is this presented and interpreted?
*unit chart timeline visualization sketches
*lit review of "timelines for analysis" papers: designs and "insights/findings"
**~LifeFlow, ~EventFlow
*dataset search: [[timelinesdb|http://www.timelinesdb.com/]], [[timelines.ws|http://timelines.ws]]
2. Misc
*consensus filtering/highlighting for multiple rankings - submit as ~InfoVis poster if Pulse paper rejected (June 26 deadline)
*Vis for tracking student progress in online education - meeting May 22 (via J Gardy)
*4YF funding suspended: Jun - Aug
*upcoming travel: New Orleans May 29 - Jun 2
*MSR internship June 8 - Sept 4
**joined ~SeaVis meetup, Google Group
*job search: emails w/ Google, Uncharted/Oculus Info, headhunter representing //"client [who] is a leading software brand with a portfolio of highly visible products for web and mobile"//
Meetings in Sept 2015:
*''Wed Sept 9'', Wed Sept 16, Wed Sept 23
1. Condorcet (MSR)
*//"Timelines Reloaded:
A Principled Design Space for Expressive Storytelling"//
*pre-paper talk Friday Sept 11 1pm
*CHI deadline Fri Sept 25 12pm
2. IEEE VIS 2015
*practice talk Tue Oct 6 10am? or Thu Oct 22 9am (I will be en route to Chicago)
*scale down from current paper talk given at UW on Fri Sept 4 (45 minutes)
*travel itinerary:
**likely Ontario on Fri Oct 9 (working 5-7 days remote on dissertation, VIS talk)
**drive to Chicago Thu 22 - Fri 23
**VIS Oct 25-30
**return to Vancouver Sunday Nov 1
*budget
**flights $~600-650 CAD (will quote price of direct YVR > ORD flights) or $500 USD
**taxi / shuttle to ORD on Nov 1 / 2 (split w/ AC)
**registration $410 USD
**accommodation (split w/ AC): $200 USD for hotel, < $200 USD for ~AirBnB x 6 / 2 = $600
**per diem $50 * 5.5 = $275 USD
**total: ~$1800 USD
3. Job search
*at MSR: met w/ DB, CW, ~StDr, ~SuDu, DF, MC, DT, RD, KI.
**interview in Jan/Feb
*Tableau meeting 08/28 w/ JM + MS
**Vancouver office has green light, will have research staff
**interview in Jan/Feb
*UW talk on 09/04, met w/ ~JHu
4. Dissertation progress
*in UBC thesis format, single bibliography
*to do:
**new intro, conclusion chapters (currently using proposal intro/conclusion)
**Task Typology chapter postscript / addendum
**preface
**redo tables to fit page format
*current aim: draft to TM by 11/15
Meetings in Nov 2015:
*Tue Nov 10* (to do: schedule individual meetings)
1. IEEE VIS 2015
*debrief
*[[VIS2015]], [[storify stream|https://storify.com/mattbrehmer/ieeevis15]]
*expense claim submitted 11/02
2. Job search
*MSR (interview in late Jan / early Feb)
*Tableau (attending open house Thu Nov 12)
*Bloomberg
*Uncharted
*Calgary
*Utah
*Harvard
3. Dissertation progress
*Draft 1 to TM Tue, Nov 17 (before CHI reviews)
*re-inserting the //how// back into the DRITW, Overview, and EMU chapters? w/ task diagrams? supplemental?
*reflection + conclusion chapter in progress
*MB to send out committee update?
*revisiting the revision/submission/defense timeline
4. CHI submission
*//"Timelines Reloaded:
A Principled Design Space for Expressive Storytelling"//
*CHI first-round reviews returned Wed Nov 18, rebuttals due Dec 1
*currently reviewing / discussing 3 CHI submissions
*fallback: ~EuroVis abstract Dec 4, full paper Dec 11; AVI in January, VIS in March?
5. Misc
*annual ethics renewed
*HL/MT visit Fri Nov 13; share interview data?
*Hacks/Hackers meetup announced Thu Nov 26
*reading week travel (3rd week of Feb)
Brainstorming/synthesis based on [[Literature Review]] (will present [[set of slides|http://dl.dropbox.com/u/6397998/wiki/11.12.08_brainstorm.pdf]])
*a methodology for investigating foo, where foo = insight/serendipity + learning + creativity
*UBC CS Research Proficiency Evaluation ([[RPE|https://www.cs.ubc.ca/students/grad/prospective/programs/phd#rpe]])
*involvement w/ [[Vismon]], AP's [[Overview]] project / [[MoDisco]]
Projects:
*DR ethnography project update: case studies
**18 user / user groups
**completed or draft: H. Lam, D. Higgins, K. Altun, S. ~Nabi-Abdolyousefi, J. D. Westbrook
**remaining: A. Saad (Torsten's ~PhD student), J. Buettgen, J. Stray
**to do: revising early summaries
**need info on J. Stray
*[[Vismon]] project (TBD)
*[[The Overview Project|http://overview.ap.org/]] (TBD)
*other project areas of interest:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
**graphical perception
[[References: Read]]: (most important of these will be discussed above in brainstorming summary)
*[[Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Andre2009 bibliography:Bibliography>> - designing for serendipity
**<<cite Klahr1999>> - methods for studying scientific discovery
*[[Information Visualization Evaluation: Meta-Analysis|]]
**<<cite North2006>> - methodologies for measuring insight
*[[Information Visualization Evaluation: Qualitative Methods]]
*[[Information Visualization Evaluation: Quantitative Methods]]
*[[Evaluation in HCI: Meta-Analysis]]
**<<cite Jennings2011>>: C&C paper, a methodology for measuring exploratory visual creativity
**CSCW (a list of 8 papers from CT - discussion of findings, but not discussion of methodology)
***<<cite Kraut1988>> - physical proximity and scientific research collaboration
***<<cite Nardi2000>> - outeraction: analyzing IM usage
[[References: To Read]]:
*[[References: To Read (Priority)]]
*[[References: VisWeek2011]]
*Creativity & Cognition proc.
*CSCW proc.
Upcoming/Ongoing:
*Moving this week
*Working remotely Dec 14-16, 19-23 (in Ottawa) - will Skype in to ~InfoVis group meeting Dec 15
**returning Wed. Jan 4 midday
*schedule next term
*courses / breadth requirement
**check out systems topic course (breadth)
**ask JP for confirmation re: breadth form
*Visiting family reading week 2012 (mid Feb - TBA)
!References
<<bibliography>>
11am - 12pm Thu
*Meeting scheduling for the term: alternating? weekly / bi-weekly? all 3 of us once monthly?
Projects:
*DR ethnography project update: case studies
**18 user / user groups
**completed: H. Lam, D. Higgins, K. Altun, S. ~Nabi-Abdolyousefi, J. D. Westbrook, A. Saad, J. Buttgen
**met w/ MS 11.12.19, reviewed work to date
**filled in summary / taxonomy table, will review w/ MS 12.01.09
**to add DR papers (i.e. <<cite Matusik2003 bibliography:Bibliography>>) to taxonomy (to discuss 12.01.09)
**TM: need info on J. Stray
*[[Vismon]] project - timeline? any feedback from Torsten?
**downloaded Vismon, completed tutorial, getting familiar w/ use cases
**reading:
***<<cite Peterman2009>>
***Vismon tech report / documentation
*[[The Overview Project|http://overview.ap.org/]] (TBD)
*interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
Courses / scheduling:
*EPSE 595 - qualitative methods readings (syllabus sent by instructor Dec 13) (began Jan 4, W 1-4pm) - unable to audit, will likely take for credit (conflict w/ MUX)
**3 textbooks, additional readings (may hold off on Charmaz text while reading these)
*systems breadth: asked JP for confirmation re: breadth form (it has been approved)
**CPSC 508 - operating systems breadth course (Fall 2012?)
**CPSC 538E - computer architecture (Greenstreet) (Spring 2012) - begins Jan 9
**CPSC 538W - online privacy (Warfield) T/R 9:30-11 - begins today
*CPSC 313 - ugrad intro to operating systems (Spring 2011 M/W/F 9-11, will sit in if possible)
*SOCI 503 (qualitative research methods, full) T/R
[[References: Read]]:
*<<cite Gigerenzer2007>> - gut feelings, hunches, intuition: the intelligence of the unconscious (book)
*[[Information Foraging, Sensemaking, Insight, & Serendipity]]: <<cite Chang2009>> - characterizing insight as both an event and a substance
*[[Information Visualization Evaluation: Qualitative Methods]]: <<cite Kang2011>> - grounded eval. of intelligence analysis process ([[VisWeek|References: VisWeek2011]])
[[References: To Read]]:
*Creativity & Cognition, CSCW proc.
*Charmaz, K. - [[Constructing Grounded Theory|http://www.amazon.ca/Constructing-Grounded-Theory-Practical-Qualitative/dp/0761973532]] - on 2hr reserve at UBC library
*D. Kahneman (2011): //Thinking Fast and Slow//
This week / upcoming:
*~InfoVis group meeting paper selection:
**from ~VisWeek: <<cite Rodgers2011>>: [[Exploring Ambient and Artistic Visualization for Residential Energy Use Feedback|http://ieeexplore.ieee.org/search/srchabstract.jsp?tp=&arnumber=6065016&openedRefinements%3D*%26filter%3DAND%28NOT%284283010803%29%29%26searchField%3DSearch+All%26queryText%3DRodgers%2C+J.+and+Bartram%2C+L.+.LB.2011.RB..+Exploring+Ambient+and+Artistic+Visualization+for+Residential+Energy+Use+Feedback.+IEEE+Transactions+on+Visualization+and+Computer+Graphics+17%2C+2489-2497.]]. IEEE Transactions on Visualization and Computer Graphics 17, 2489-2497. (should be a good mix of design requirements, aesthetics, evaluation, casual/ambient vis)
**alternatively: <<cite Mayr2010>> [[BELIV '10 paper on evaluating problem solving|http://www.beliv.org/papers/beliv2010/Research%20Papers/8-BELIV_problemsolving_2010-02-12_v2.pdf]], later a IV journal paper (Jul 2011 special issue) - latter [[available only as draft|http://www.donau-uni.ac.at/imperia/md/images/department/wissenkommunikation/news/mayr-smuc-risku_ivs-2011_pre-print.pdf]] (UBC library 1year publication delay)
*CHI paper accepted! some revisions needed for camera-ready Jan. 13
*Visiting family (tentatively Feb 9 - 19)
!References
<<bibliography>>
11am - 12pm Thu
*Meeting scheduling for the term: alternating? weekly / bi-weekly? all 3 of us once monthly?
**No regular meeting next week (attending workshop)
Projects:
*[[Vismon]] project
**timeline & scope: longitudinal study
**goals & contributions, research questions, focus: usability, efficacy (decision making, communication), learnability, insight
**identifying stakeholders
**downloaded Vismon, completed tutorial, getting familiar w/ use cases
**reading:
***<<cite Peterman2009 bibliography:Bibliography>>
***Vismon tech report / documentation
*HDD taxonomy project
**completed summaries for each case study
**existing taxonomy summary table filled in
**this week: additional taxonomy brainstorming: will meet w/ MS tomorrow 10am
**next week: case study write-ups
*Other potential projects:
**[[Overview Project|http://overview.ap.org/]]
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
Courses / scheduling:
*EPSE 595 - qualitative research methods: data collection and analysis (Wed 1-4pm)
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
*systems breadth: asked JP for confirmation re: breadth form (it has been approved)
**CPSC 508 - operating systems breadth course (Fall 2012?)
*sat in on first session of:
**SOCI 503 (qualitative research, seminar-based, highly-theoretical)
**CPSC 538E - computer architecture (Greenstreet)
**CPSC 538W - online privacy (Warfield)
[[Reading recently|References: Read]]:
*<<cite Crotty1998>>: foundations of social research (EPSE 595 text)
*<<cite Gigerenzer2007>> - gut feelings, hunches, intuition: the intelligence of the unconscious, decision-making under uncertainty
[[References: To Read]]:
*Creativity & Cognition, CSCW proc.
*Charmaz, K. - [[Constructing Grounded Theory|http://www.amazon.ca/Constructing-Grounded-Theory-Practical-Qualitative/dp/0761973532]] - 2hr reserve at UBC library
*D. Kahneman (2011): //Thinking Fast and Slow// (addressing the same issues as <<cite Gigerenzer2007>>, however taking a different stance)
This week / upcoming:
*CHI paper
**revisions for camera-ready Jan. 13
**revision statement
**contribution statement
*Next week: GRAND workshop at SLAIS on text analysis (Thursday AM)
*plans to visit family mid-Feb uncertain
!References
<<bibliography>>
10:30-11:30am Thursday
*Meeting scheduling for the term: week 1: flex/no meeting, week 2: TM +JM (away this month), week 3: TM (this month +JM), week 4: JM
''The RPE'':
*Info session 12.02.04 (presentation by Wolfgang followed by Q & A)
*4 month project
**possibility of counting M.Sc research as RPE ^^1^^
*12p report submitted by Sept. 15 ^^2^^
*Early October oral presentation to RPE committee (TM, JM, + another faculty member we recruit, who doesn't necessarily have to be in CS).
**report can NOT be co-authored with supervisor(s)
**RPE supervisory committee must be selected by May 1
*contribution to the [[HDD-DR Ethnographic Project]] w/ MS would NOT count as an RPE project ^^3^^
Projects:
*[[Vismon]] project
**[[Vismon: Research Questions]], initial response from Torsten, Randall, our position,
**R. Peterman's response:
>//don't take my opinions as necessarily correct. Instead, I would suggest that you go ahead and ask the same questions of Mike Jones to see what he thinks. He might have quite a different view of the matter. [...] I have already found some potential "champions" by exposing many people to [[Vismon]]. [...] Perhaps we should discuss this further in a conference call or a meeting.//
*[[HDD-DR Ethnographic Project]]
**filled in our Q&A taxonomy table with use cases
**preliminary case study selection
**meeting w/ MS tomorrow to discuss next steps, roles
*[[Overview Project|http://overview.ap.org/]] - data journalism - what are users actually doing?
**tasks, data, tools, processes, collaboration
**hypothesis generation, validation
**using insight-based evaluation methods in this domain
**J. Stray visiting Vancouver in 12.03.05
*Other potential projects:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique and/or extending to HD to HDD/DR data
***to do: contact author, find out if anyone is validating the method
***read Heer, Kosara papers re: Mechanical Turk
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**epistemologies: objectivism vs. constructionism vs. subjectivism (<<cite Crotty1998>>: foundations of social research text)
***positivism, interpretivism (symbolic interactionism, phenomenology, hermeneutics), critical inquiry
**reading qualitative research (Pascoe 2007: a case study)
**the [[ethics|Research Ethics]] of qualitative research
**<<cite Silverman2007>>: short, interesting, cheap book about qualitative research
***tiny details matter: finding the extraordinary amongst the seemingly mundane, and conversely finding the mundane amongst the seemingly extraordinary
***manufacturing vs. finding data
***sequences vs. instances
***conversation/discourse analysis
***on bullshit
**[[grounded theory|Grounded Evaluation]]
**[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
**assignment 2: participant observation, field notes
**upcoming: interviewing, focus groups. found data, data analysis and representation
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
**next week: catching up on missed lectures
[[Reading recently|References: Read]]:
*<<cite Wickham2010>> - graphical inference
*<<cite Newell1985>> - prospects for psychological science in HCI
*<<cite Becker1957>> - participant observation vs. interviewing (for EPSE 595)
[[References: To Read]]:
*Hayes' Action Research in HCI (referred to by MS)
*Fontana & Frey (1994) on the Art of Interviewing (for EPSE 595)
*continue reading Creativity & Cognition, CSCW proc.
Upcoming:
*today: demo (~C-TOC) and lunch w/ J. Birnholtz, DLS
*3 GRAND reviews to complete for next week
*CHI submission
**fixed author affiliation typo, ordering
**registration (Mar 13)
**CHI madness preparation (Mar 23)
*qual. methods course talk to MUX in April / May
*RPE timeline
*Internship positions (Winter / Summer 2013) - e.g. Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
**other ideas?
!!RPE Notes
^^1^^ Grad affairs' argument is that middle-author contributions are hard to tease out, and could not serve as an accurate gauge of independent research potential in the same capacity that an independent RPE project would.
^^2^^ As this is the RPE's first year of existence, there are a few students in the program, such as myself, who completed M.Sc degrees at UBC in CS, and are continuing on to do a ~PhD with the same supervisor (or in my case, one of the same supervisors). Laks says that on a case-by-case basis, these students' M.Sc thesis topics could be considered as an RPE project. In future years, this possibility would be phased out and replaced by the ~PhD-track M.Sc program. Apparently there was some prior faculty committee discussion of this, which Wolfgang was unaware of. He seemed resistant to the idea. We'll need to discuss with Grad Affairs should we pursue this option. [I'm not sure if switching research areas between my M.Sc and ~PhD will kill this option. I'll have to ask, as I just thought of this while writing.]
^^3^^ re: what gets done by Sept 15. The RPE project is designed be a unit of research that fits in with the eventual thesis topic. It's not necessary that the RPE project represent a completed body of work. It should, however, be a sizeable chunk. For instance, while completing and analyzing the results of a full user study, with dependencies on participant recruitment and/or collaborator availability, may be unfeasible if it is to be finished by Sept. 15. However, a good part of the study design, piloting, and data collection may get done by then.
11am Thursday
*Meeting scheduling for the term: week 1: flex/no meeting (TM), week 2: TM +JM, week 3: (TM conflict, JM instead), week 4: (JM conflict, no meeting)
Projects:
*[[Overview Project|http://overview.ap.org/]] - data journalism - [[J. Stray|http://jonathanstray.com/]] debriefing 12.03.05 visit
**[[Overview Discussion 12.03.05]]
***//who are the users?// (none yet); intended: professional journalists (primary), DIY hacker types (secondary); JS not a representative user - sees himself as on the fence b/w developer and journalist
***//where are the users?// Newsrooms, some at AP
***//what are users currently doing?// Document Cloud, Keyword search, manipulating spreadsheets
***//what can Overview do that previous tools cannot?// Orientation, sense making, exploration of dataset structure, an *overview*
***//how we can evaluate Overview?//
**2 studies being run in parallel:
#post-deployment field data collection from real users regarding usage patterns, context of use: email, phone/skype
#insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
*to do:
**generate deployment survey Qs for #1, iterate w/ JS, TM
**brainstorm methodology for #2
**project timeline and the RPE, determine supervisory committee
*[[Vismon]]
**[[Vismon: Research Questions]], further response from Randall, Torsten
*[[DR in the Wild|HDD-DR Ethnographic Project]]
**taxonomy colour coding: fail/struggle/happy usage patterns
**methodology token - describing what we did, received feedback from EPSE 595 prof
**meeting w/ MS tomorrow
*Other potential projects:
**interruptions and multi-tasking + ~InfoVis/VA (Decision making, collaboration)
***could tie in w/ data journalism work - research Qs addressing context of use
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>) validating the technique, then possibly extending to HD to HDD/DR data
***to do: contact author, find out if anyone is validating the method - nothing published so far
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**[[Interviewing]] - social constructivist perspective, structured, open-ended, post-modern, [[Group Interviews]]
**[[grounded theory|Grounded Evaluation]]
**[[Ethnography]] - MIT anthropologists studying marine biologists in the wild
**[[PhotoVoice|http://prezi.com/_m_lndsuctib/photovoice/]]
**[[assignment 1|EPSE 595 Workbook 1]]: reflecting on my personal epistemology, conceptual maps, interpretive and critical research questions (on data journalism - what are users actually doing?)
**[[assignment 2|EPSE 595 Workbook 2]] participant observation, field notes
***[[assignment 3|EPSE 595 Workbook 3]] interviews (JS visit wouldn't count)
**upcoming: focus groups. found data, data analysis and representation
*CPSC 313 - ugrad intro to operating systems (M/W/F 9-10am), sitting in
[[References: To Read]]:
*Hayes' Action Research in HCI (referred to by MS)
*continue reading Creativity & Cognition, CSCW proc.
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Upcoming:
*CHI submission
**registration (Mar 13)
**CHI madness preparation (Mar 23)
*qual. methods course talk to MUX in April / May
*RPE timeline: see [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
**other ideas?
11am Thursday
*Meetings in April: 04/05: flex meeting (JM), ''04/12: (TM +JM)'', 04/19: (TM), 04/26: (JM - last meeting before GRAND / CHI)
Projects:
*[[Overview]] Project - see [[notes from meeting w/ JS|Overview Discussion 12.03.05]]
**2 studies being run in parallel:
***1. [[Document mining in data journalism]] - post-deployment field data collection from users - [[EPSE 595]] final project proposal (submitted Apr 11),
****Finalizing [[interview foci and questions|Overview Deployment Field Survey]]
****User #1: Tulsa World reporter investigating missing municipal funds allocated to police equipment purchases, 16K email corpus - interview not yet scheduled,
*****''to do'': follow-up w/ JS - piloting, coordinating interviewer roles
****Pilot interview w/ JS, find out split of responsibilities
****JS reference: [[Profiles of the Data Journalist|http://radar.oreilly.com/2012/03/profile-of-the-data-journalist.html]] by [[Alex Howard / @digiphile|https://twitter.com/#!/digiphile]]
***2. insight-based study with data journalism students (Columbia, ~McGill, Hong Kong - JS has connections)
****to do: brainstorm methodology, follow up w/ JS re: involvement of faculty and students
**''RPE'': determine supervisory committee, see [[notes from 12.02.16 meeting|TM-JM-12.02.16]]
***''to do'': ask RR; other possibilities: GM, SM. Long-term ~PhD committee contenders: SC (Calgary), MT (Victoria)
**[[LeakExplorer|http://www.leaksplorer.org/]] - a related project out of U of T
**possible involvement of CW Anderson (Asst. Prof Communications, CUNY), newsroom ethnographer (see forwarded email)
***''to do'': contact CW re: collaboration
***[[Truth, documents and data journalism鳴ory|http://www.reporterslab.org/cw-anderson/]], [[Reporters' Lab|http://www.reporterslab.org/]]
***[[The Things That Tell Us Whatⵥ (a Little Research Manifesto)|http://journalismschool.wordpress.com/2011/03/11/the-things-that-tell-us-what-is-true-a-little-research-manifesto/]]
***Anderson, C. W. and Kreiss, D. (2013). Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism.
**software:
***JD on recording Skype calls: [[ECamm Call Recorder|http://www.ecamm.com/mac/callrecorder/]]
***[[HyperRESEARCH, HyperTRANSCRIBE|http://www.researchware.com/products/hyperbundle.html]] - used in EPSE 595
***[[dedoose|http://www.dedoose.com/]] - J. Woefler / CT recommended
**~MoDisco to become tech report, JS to blog sections of it
*Other projects:
**Graphical inference (<<cite Wickham2010 bibliography:Bibliography>>), controlled lab study to validate the technique, then extend to HDD/DR data
**CT follow-up journal paper on interruptions work - will talk to CT re: commitment, level of involvement Friday 1pm
Courses:
*[[EPSE 595]] - qualitative research methods: data collection and analysis (Wed 1-4pm)
**Recent Topics:
**[[Organizing and Making Sense of Data]]
**[[Computer Assisted Data Analysis, Data Displays]]
**[[Representing Knowledge]]
**Assignments:
***[[assignment 5|EPSE 595 Workbook 5]] preliminary data analysis
***final project: interpretive / critical project proposal: [[Document mining in data journalism]]
Reading:
*continue reading <<cite Charmaz2006>> - Grounded Theory
*Anderson, C. W. and Kreiss, D. (2013). Black-boxes as capacities for and constraints on action: ANT and ethnography of electoral politics and journalism.
*<<cite Frankfurt2005>> - cited in <<cite Silverman2007>>
To Read:
*<<cite Sprague2012>> - ~InfoVis group meeting paper for tomorrow
*Confessions of a Grounded Theory ~PhD (Furniss, CHI 2011)
*Heer, Kosara papers re: Mechanical Turk (for Graphical inference project idea)
*King & Grimmer paper on clustering
*O'Brien paper on use of insight evaluation
Upcoming:
*CHI practice talk at MUX, April 25 - to do: prepare this presentation
*In Texas @CHI May 6-15
*Internship positions (Winter / Summer 2013) - e.g. AP, Adobe's Advanced Technology Labs, Tableau, Google, MSR, others?
**will chat w/ people @ CHI
!!References
<<bibliography>>
11am Thursday
*Meetings in June: ''06/07: TM'', 06/14: TM+JM, 06/21: TM+JM, 06/28: TM+JM
[[Overview]] Project
*''Ethics'': changing names and titles on consent forms (likely OK) - no ethics info changed
**''to do'': contact UBC BREB re: remote interviews - Shirley Thomson (sp?)
**MB boosted to 1.3 clearance.崨ics for ~JSchool student project TBD when JS's teaching post / ability to conduct study confirmed
**amendment needed for Overview study w/ jSchool students
*to schedule follow-up debriefing w/ JS and TM
*created SVN project today
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[interview foci and questions|Overview Deployment Field Survey]]
*User #1: JW: Tulsa World reporter investigating emails re: police equipment purchases
**[[JW's story|http://tulsaworld.com/tpdtechemails]]
**[[project on DocumentCloud|http://www.documentcloud.org/public/search/projectid:%205269-tpd_emails]] - only 125 emails shared by Tulsa World
**JW sent overview_log, tag file, source data CSV, loading into Overview, excellent notes
**49 reader comments, some relevant to analysis methods used by JW
**~1h40min interview 06/06/12 via G+ hangout, screen sharing, recorded video and audio w/ ~IShowUHD
**follow-up via email, expecting consent form to be returned
**completed transcription (around a 8:1 transcription:interview time ratio) (13,000 words / ~250 statements)
*Other users' status? - keeping track in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
2. [[An Insight-Based Evaluation of Overview]] with data journalism students
*to discuss w/ JS re: part-time instructor post this fall
Using Overview (and other document mining tools, techniques, scripts):
* [[Visually exploring 13,000 music reviews with Overview|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]] - JS says this is the largest dataset used so far in Overview
*Overview w/ Vancouver city council meeting minutes (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]) (~2,500 pages)
*~2,000 news articles from various media outlets relating to Vancover's DTES (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]])
Overview misc:
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: http://lensingwikipedia.cs.sfu.ca/
**meeting Jul 25 4:30pm (following ~InfoVis)
*email thread w/ K. Melnick, [[VIVA VA challenge group|https://sites.google.com/site/challengeva/]] member re: Overview - likely to give a demo to big group at some point, discussed Vancouver Sun collaboration
**also collaborating w/ Anoop
**to meet w/ VA challenge group Monday 4pm
**VA Challenge program workshop June 28 on structured data analysis
**CANVAS: canadian visual analytic summer school Jul 23-25
*met K. Rozendal, Vancouver Sun reporter covering Saul's visit, interested in Overview; also a UBC ~J-School student - interested in using Overview with Vancouver city council meeting minutes
>//"I think seeing your pre-processing strategies and the tools you used would be a nice element of a demo. At the moment, there's not much I could do with either the recipe or the final dish of docs, but maybe we can sit down later in the summer for a meeting where I can take a look. What sort of blocks on your time do you have in the next few weeks? Given the local and political nature of the content, the minutes would be the perfect raw material for showing some of us at the j-school a bit about visual analytics.//"
*compiling [[Data Journalism: Links and Literature]] (from JS, CWA)
*email conversation w/ CWA @ CUNY (Communications), newsroom ethnographer
*(re)read Wattenberg's //Baby Names// and Heer's //sense.us// papers: users in the wild doing not what we expected
*read DSM review re: griping on [[Grounded Theory]]
[[Graphical Inference User Studies]] Project
*[[compiled list of papers|Graphical Inference Evaluation]] citing / extending original graphical inference paper
*read <<cite Willett2012 bibliography:Bibliography>>
*''to do'': read Heer, Kosara, et al Mechanical Turk papers
[[recently read|References: Read]]:
*<<cite Jianu2012>> - small UI changes affect analytic strategies (CHI 2012): [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*<<cite Charmaz2006>> - Grounded Theory ref (ongoing)
[[to read|References: To Read]]:
*Heer, J. and Shneiderman, B. (2012). Interactive Dynamics for Visual Analysis. Communications of the ACM.
*[[References: CHI2012]]
Upcoming / ongoing:
*SIGGRAPH ASIA review request, July 13 - sanity check to TM
*DRITW rejected from ~InfoVis - fast tracked to TVCG - next steps?
**to meet w/ TM, MS, SI (2 weeks from now after ~InfoVis revisions submitted for DSM, ~RelEx)
*~ToCHI journal paper on domestic interruptions with CT, JM
**TOCHI In The Wild special issue July 2 deadline
*vacation middle week of July (13, 16-17, 20) / beginning of August (July 26 - Aug 9)
!!References
<<bibliography>>
11am Thursday
*Meetings in July: JM+TM: 07/19, 07/26
[[Overview]] Project
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[ongoing analysis presentation|https://dl.dropbox.com/u/6397998/docmining_analysis_07.19.pdf]]
*Other users' status? - keeping track in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
*I plan to share current results and check in w/ JS following this meeting
*Next steps: refining the [[interview foci and questions|Overview Deployment Field Survey]]
**integrating additional tagged data from extant texts: PBS blog post, process log, correspondence thread
*ideas about personal information management in visualization, incremental tagging vs. one-shot en-masse tagging, structured vs. unstructured information
2. [[An Insight-Based Evaluation of Overview]] with data journalism students
*JS has part-time instructor post
*web-based Overview v2 within 6 months
*study more likely to occur in Winter 2013 with v2
other Overview-related "thingies":
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: [[Lensing Wikipedia|http://lensingwikipedia.cs.sfu.ca/]]
**meeting Jul 25 4:30pm (following ~InfoVis)
*met with [[VIVA VA challenge group|https://sites.google.com/site/challengeva/]] Monday Jun 25: demo and Q+A
**interested, no one currently doing unstructured data analysis
**also collaborating w/ Anoop
**[[CANVAS|http://www.viva-viva.ca/newsandevents.html]]: canadian visual analytic summer school Jul 23-25 (JD Fekete keynote)
*met K. Rozendal, Vancouver Sun reporter, interested in Overview; also a UBC ~J-School student - interested in using Overview with Vancouver city council meeting minutes
Using Overview (and other document mining tools, techniques, scripts):
* [[Visually exploring 13,000 music reviews with Overview|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]] - JS says this is the largest dataset used so far in Overview
*Overview w/ Vancouver city council meeting minutes (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]) (~2,500 pages)
*~2,000 news articles from various media outlets relating to Vancover's DTES (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]])
Recent and ongoing:
*thoughts on MUX work environment (or lack thereof)
*SIGGRAPH ASIA review
*DRITW rejected from ~InfoVis - fast tracked to TVCG - MS has token, next steps week of Aug 13
*~ToCHI journal paper on domestic interruptions with CT, JM submitted Jul 17
*reviewing TM textbook ch. 13
*reading gleicher write-ups on [[phd theses|http://pages.cs.wisc.edu/~gleicher/Web/Advice/PrelimsAndThesis]] and [[status reports|http://pages.cs.wisc.edu/~gleicher/Web/Advice/StatusReports]]
*vacation: weeks of Jul 30, Aug 6
*@TM: opportunities for studying unstructured text/document mining at Twitter?
!!References
<<bibliography>>
*Meetings in Aug: ''JM+TM: 08/16'', TM+JM: 08/23, TM+JM: 08/30
*thoughts on x508 ~MUXlab
[[Overview]] Project
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*met with JS 07.26
*Development update: web-based Overview Alpha, developed in Scala, deployment in September at Online News Assoc, conference (not as tech-centric as NICAR)
**Document Cloud barrier less of a restriction than current Ruby/config/install barrier (eventually Document Cloud constraint will go away)
**new tree layout: collapse/expand nodes, pruning by node size, pruning by depth; node size proportional to number of documents in node; panning and zooming; only active tag colours nodes
***explicit (expand/collapse) vs. implicit (screen real estate) tree interaction
**discussed implications for logging
*Overview Users
**keeping track of users' status in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**6 pending, 2 completed (+1 pilot), 3 aborted
***pending: (nothing since 07.27), journalist with Swedish govt. emails most recent
*Discussion:
**ideas about personal information management in visualization, incremental tagging vs. one-shot en-masse tagging, structured vs. unstructured information
**user goals: finding the smoking gun (read most/all documents) vs. finding the major trends (read some)
**could reflect difference between document requests (Եlsa PD) vs. document dumps (ɲaqi war logs)
**how to deal with weird / unexpected documents
*Reflections:
**project to take back-burner role until either (a) a user finishes a story with current version or (b) Overview online alpha is released in late September
**current data analysis process overkill for design purposes; with additional users, some interesting usage patterns might emerge; nevertheless, JS ecstatic to have rich user data
**future interviews: worth transcribing? worth doing log file analysis? perhaps only if usage pattern is suspected to differ substantially from existing users (decide on-the-fly)
***implications for paper writing: re: consistency of results, can't call it [[Grounded Theory]], more likely adopting a case study approach
*Next steps:
**refining the [[interview foci and questions|Overview Deployment Field Survey]] (done)
**writing RPE pre-paper talk, RPE paper
2. [[An Insight-Based Evaluation of Overview]] with data journalism students
*study more likely to occur in Winter 2013 with Overview 0.1 Alpha
''New projects'' and collaboration and/or internship placement opportunities
*Mixed-method, in the wild, evaluation of visualization tools for supporting exploratory data analysis of high-dimensional data
**Ideally avoiding the gatekeeper / limited access to users scenario (϶erview)
**Local domain scientists seeking collaboration?
**Industry contacts seeking HCI interns to assist in the evaluation of vis. tools (TM contacts at AT&T? Tableau?)
*Crowd-sourcing evaluation of vis. tools/techniques for exploratory data analysis of high-dimensional data
*Extending <<cite Wickham2010 bibliography:Bibliography>>'s graphical inference methods to ~SPLOMs, DR data, follow-on from DRITW
Reading:
*<<cite Heer2010>>: [[notes|Information Visualization Evaluation: Meta-Analysis]] - crowdsourcing graphical perception
*<<cite Kairam2012>>: [[notes|Information Visualization: Techniques]] - ~GraphPrism AVI paper, use of <<cite Wickham2010>>'s Graphical Inference
*<<cite Pinaud2012>>: [[notes|Information Visualization: Design Studies]] - design study on graph re-writing, nice task taxonomy in nested model framing
*<<cite Ziemkiewicz2012>>: [[notes|Information Visualization Evaluation: Qualitative Methods]] - observing different visualization use strategies among similar users
*<<cite Wood2011>>: [[notes|Information Visualization: Design Studies]] - ~BallotMaps (for ~InfoVis reading group), use of <<cite Wickham2010>>'s Graphical Inference
*<<cite Heer2007a>>: asynchronous collaboration and visualization: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]] - social data analysis patterns / sense.us
*<<cite Wagstaff2012>>: ML that matters - [[notes|Information Visualization Evaluation: Meta-Analysis]] - "~InfoVis that matters"
*<<cite Marchionini2006>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]] - characterizing types of search: lookup vs. exploring: learning and investigating
*<<cite Grimmer2011>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]] - meta-clustering, case-studies and mixed-method eval.
*<<cite Budiu2009>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]] - comparing different tagging interfaces and memory for tagged items
*<<cite Jianu2012>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]] - small UI changes affect scientific data analysis strategy
*<<cite Willett2012>>: [[notes|Information Visualization Evaluation: Qualitative Methods]] - strategies for social data analysis
*Gleicher write-ups on [[phd theses|http://pages.cs.wisc.edu/~gleicher/Web/Advice/PrelimsAndThesis]] and [[status reports|http://pages.cs.wisc.edu/~gleicher/Web/Advice/StatusReports]]
>Your report should have the following following 6 sections:
>>1. What were my goals for this week?
>>2. What did I accomplish this week?
>>3. Why are numbers 1 and 2 different?
>>4. What are my goals for next week?
>>5. How does this fit in to my bigger picture? How does #4 fit into my longer term goals? (What are the longer term goals?) What deadlines are looming beyond the week horizon?
>>6. What did I read this week?
Upcoming / to do:
*continue brainstorming graphical inference project
*September - Dec meetings, tentatively Wed 11:00 - 12:30
*RPE paper Sept 15
**highly reflective, not claiming contributions, lower level of detail, aim for 6p
*Research pitch to Tableau (end of August)
**research interests, background, project ideas
>''abstract'': Mixed-method evaluation of visualization tools and techniques
>''current interests'': Exploratory data analysis, serendipitous discovery; High-dimensional data, DR data, unstructured text data
>''emerging methodologies'': Crowdsourcing social data analysis, graphical inference
>''ongoing interest'': Graphical Perception, Visual Attention, ~Task-Switching and Interruptions
>''background'': HCI M.Sc specialization supervised by J. ~McGrenere, Cognitive Science undergraduate degree, 16 month professional internship as UX designer at EMC
>''courses'': in quant. and qual. research methods, visualization, HCI, visual display design and graphical perception, scene perception, data mining
*~VisWeek 2012 registration (ASAP)
**ACM student membership renewal
*DRITW next steps meeting (date TBD, week of Sept 15)
*Upcoming ~InfoVis slot (Sept 5 - <<cite Willett2012>>?
*EPSE 595 pitch at upcoming MUX meeting
*orientation committee work (first week of Sept.)
*taking CPSC 508 (operating systems) this fall, will complete ~PhD course requirement
!!References
<<bibliography>>
*Meetings in Aug: JM+TM: 08/16, ''TM+JM: 08/30''
[[Overview]] Project
*RPE paper draft - thoughts / comments?
*writing as thinking in mixed-methods constructivist research very helpful
*to add?: planned study w/ journalism students; my own use of Overview; RW in Vis + log file analysis; missing RW?
*recent relevant reading:
**<<cite Beaudouin-Lafon2004 bibliography:Bibliography>> - interaction models for evaluative, generative, evaluative power
**<<cite Pohl2010>>: [[notes|Information Visualization Evaluation: Qualitative Methods]] - Prov. BELIV '10, log file analysis of EDA
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*Overview Users
**keeping track of users' status in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**8 pending, 2 completed (+1 pilot), 3 aborted
***pending: molecular/cellular bio researcher with paper abstracts (08/29), York U communications researcher on ~WikiLeaks (08/27)
2. [[An Insight-Based Evaluation of Overview]] with data journalism students - make mention of this in RPE paper?
*study more likely to occur in Winter 2013 with Overview 0.1 Alpha
''New projects'' and collaboration and/or internship placement opportunities
*Ideally avoiding the gatekeeper / limited access to users scenario (϶erview)
3. Tableau internship potential 2013 - upcoming phone/Skype call w/ J. Mackinlay (Sept)
*Mixed-method evaluation of visualization tools/techniques for supporting exploratory data analysis
>My research is about furthering methodologies for evaluating visualization tools and techniques. This involves executing, comparing, and writing about quantitative and qualitative methods used for determining whether a visualization tool or technique is usable, useful, and appropriate for its deployment context. Another aspect of this work is determining at which points these methods are appropriate and informative during the various phases of tool or technique development. Finally, this work will allow me to explore the potential of emerging evaluation methods, such as those presented at recent "Beyond Time and Error" novel evaluation (BELIV) workshops.
>
>Of particular interest to me is how we can design evaluation methodologies that assess how visualization tools support exploratory data analysis and serendipitous discovery. My most recent project is an example of this: a mixed-method evaluation of Overview, a tool for visualizing and exploring large text document sets, built by our collaborators at the Associated Press. My ongoing objective will be to compare the efficacy of evaluation methods used in the Overview project with those used in future projects.
>
>I came to be interested in this area or research during my undergraduate studies in cognitive science, where I was fascinated by graphical perception, working memory, and attention, along with their implications for interaction design. This interest persisted throughout my Master's work in HCI, where I focused on issues relating to task-switching and interruptions. It was during this time that I began to realise the limitations of quantitative methodologies for answering many research questions, and I acknowledged a need for the triangulation of research methods, both quantitative and qualitative, those deployed in controlled settings as well as those deployed "in the wild".
*JM:
>//"I don't think Tableau is very interested in an investment in formal evaluation methodologies because we are having tremendous success with informal evaluation. We are one of those lucky companies that enthusiastically use the software we create. Nevertheless, I would be delighted to talk to Matt."//
4. Extending/validating <<cite Buja2009>> and <<cite Wickham2010>>'s graphical inference methods
*recent relevant reading:
**<<cite Majumder2010>> - Iowa State Tech Report on graphical inference, used for regression model parameters; [[MTurk|http://mturk.com/]] study
**<<cite Zhao2012>> - Iowa State Tech Report on graphical inference (eye-tracking study)
**<<cite Kairam2012>>: [[notes|Information Visualization: Techniques]] - ~GraphPrism AVI paper, use of graphical inference, also used in <<cite Wood2011>>'s ~BallotMaps paper
*extending method to ~SPLOMs / <<cite Sedlmair2012>>, DR data, follow-on from DRITW: implications for more than 20 plots, p values
*teaching myself R, ggplot (<<cite Wickham2010b>>, qplot, nullabor (<<cite Wickham2010>>)
*Crowd-sourcing evaluation of vis. tools/techniques for exploratory data analysis
**<<cite Heer2010>>: [[notes|Information Visualization Evaluation: Meta-Analysis]] - crowdsourcing graphical perception
**<<cite Kosara2010>>: [[notes|Information Visualization Evaluation: Meta-Analysis]] on conducting ~InfoVis evaluation studies with [[MTurk|http://mturk.com/]]
**<<cite Willett2012>>: [[notes|Information Visualization Evaluation: Qualitative Methods]] - strategies for social data analysis
Upcoming / to do:
*September - Dec meetings, Wed 11:00 - 12:30 (1st / 3rd week TM + JM or JM, 2nd or 4th week: TM)
*~VisWeek 2012 registration (done) (Oct 14 - 19) - BELIV workshop Oct 14/15
*ACM student membership renewal (done)
*DRITW next steps meeting (date TBD, week of Sept 15)
*~InfoVis group website (Sept 10)
*EPSE 595 pitch at upcoming MUX meeting? or RPE presentation? TBD
*opportunities for collaboration in Germany in 2013? (Deutscher Akademischer Austausch Dienst grants - Oct 5)
*MUX work party tomorrow
*orientation committee work (first week of Sept.) (mostly done)
*taking CPSC 508 (operating systems) this fall, will complete ~PhD course requirement
!!References
<<bibliography>>
*Meetings in Nov: ''TM+JM 11/07'', TM 11/14, JM 11/21, (TM 11/28, if necessary)
1. ~VisWeek induced identity crisis
*Or, a mismatch between my current research interests, the interests of industry, my career interests� messages:
>a senior researcher in academia: //"you're going to do a whole ~PhD on ''Evaluation''?//
>
>a senior researcher in industry: //"that sounds like a ~PhD!"//
*On Tableau
**not interested in research on evaluation, tasks, exploratory data analysis:
>//"the purpose of Tableau is speed, not insight"//
>
>//"we know what the tasks are"//
**instead: how do users tell stories with data? maintain objectivity? control for cognitive bias? what makes users migrate from Tableau public to Tableau desktop? mining logged usage data, no lab/field user studies
**incrementally adding support for unstructured data, network/graph data
*My thoughts on [[BELIV|BELIV-12: Notes]] / [[VisWeek|VisWeek12 Notes]]
**BELIV provided a better vocabulary, but no new ideas: a saturation point?
**attended some interesting industry panels:
***largely no need for new interactions/visual encodings, happy with off-the-shelf tools and techniques
***Bloomberg talk on designing custom workstations for financial analysts
**to read: <<cite Zhang2012 bibliography:VisWeek2012>> on comparing commercial systems
2. The ~PhD, one year in� on track?
*Reflections on projects central, peripheral, and abandoned: Overview, DRITW, Task Taxonomy, Vismon
*Have I been productive or just busy?
*Is this normal?
3. Making a case for the HCI ~PhD breadth requirement
*struggling with systems, a time vacuum
*(not sure if this is due to poor time management, low motivation, a poor understanding of the material, or some combination of these)
4. [[Task Taxonomy|Task Characterization: Meta Analysis]]
*Slipped from Dec. ~EuroVis to March ~InfoVis
*Starting to converge, thinking about methodology
*BELIV discussion: arguments for/against a mid-level task taxonomy
*Added some ~VisWeek papers to the to read queue:
**<<cite Crouser2012 bibliography:VisWeek2012>> - on affordances in Vis/VA
**<<cite Pohl2012a>> - comparing high-level theories describing VA process
**<<cite Cottam2012a>> - on describing dynamic vis
**~McNamara BELIV position paper on Mental Model Formation through Information Foraging: <<cite Mcnamara2012>>
*Meeting w/ TM 11/14
5. Overview
*AP journalist used Overview to mine Paul Ryan's correspondence
*Story published in mid Oct, sent us log files and dataset, will be setting up an interview soon (now that the election is over)
*Still need to finish a pitch to UBC journalism / humanities via RR
6. DRITW
*MS and I have revised the ~InfoVis submission, now a 14p TVCG paper
7. Graphical inference
*A ~VisWeek paper validates the method in a large-scale user study: <<cite Hofmann2012a>>
*potential follow on work that addresses DRITW / SPLOM work: spotting true/false positives/negatives
8. Other recent goings-on:
*Read a paper on random projections for ~InfoVis group meeting (<<cite Anand2012>>)
*Met Liane Gabora (UBC Psych.), Vicki Lemieux (UBC SLAIS / MAGIC)
*Gave a guest lecture to CPSC 344 (CHI talk)
*Wrote a CHI review
*@JM: CTOC visiting student
!References
<<bibliography VisWeek2012>>
*Meetings in October: 10.03, 10.10, 10.23, ''10.30''
1. Overview
*JS' student study design document (email link to Google Doc)
*~EuroVis paper:
**log files sent by SA (Texas); JS should have access to documents, tags
**recently reading: <<cite Gorg2013 bibliography:Bibliography>> TVCG on Jigsaw, <<cite Dou2013a>> VAST '13 on Hierarchical Topics, <<cite Liu2013>> on Newdle
**to do: factor in revisions based on paper draft comments
*CPSC 344/544 Guest lecture (see below)
2. Pulse
*started 10.28
*recently: interviewed MT (Surrey SB) 10.24, NV (SES) 09.18
*ongoing consolidating interview data: [[notes from KN (UCB) interview|Pulse-KN-13.08.28]], [[notes from JS (McGill) interview|Pulse-JC-13.07.29-MEB-13.08.03]]
**to do: user requirements document, 2nd round interviews
*upcoming: interviewing JC (Pulse), energy manager from BC retirement communities
*likely to give an informal talk about my research / background at some point over the next couple of weeks
3. Misc.
*I'm giving a 35 min guest lecture in CPSC 344/544 Nov 14 (9:30 am): "//Where HCI meets Visual Analytics, Document Leaks, and Investigative Journalism//", abstract:
>Over the past few years, there have been a number of high-profile leaks of sensitive collections documents, such as the leak of hundreds of thousands of top-secret US State Department diplomatic cables to ~WikiLeaks in 2011. When investigative journalists gain access to these document collections, which are often comprised of unstructured text, it's hard for them to know where to begin: a document collection might contain several newsworthy stories, but finding them is difficult through keyword search alone. Since 2010, the UBC Information Visualization research group has collaborated with the Associated Press, the Columbia Journalism School, and a number of investigative journalists toward the design and evaluation of Overview (http://overviewproject.org), a web-based visual analytics application for journalists that allows them to systematically explore large and messy text document collections, helping them to find recurring patterns and anomalous documents. I'll talk about where HCI fits into this ongoing project, addressing questions such as: (1) What makes a visual analytics application effective? (2) What are some of the challenges of working with external collaborators and subject matter experts? (3) How do you study user adoption after you've released an application to the web? and (4) How do you design for infrequent or one-time use by users with a range of technical backgrounds?
*1 CHI review due Nov. 5
*recent: meetups
**[[Data Science, Machine Learning, Marketing, and the Customer Experience|http://www.meetup.com/DataScience/events/142278212/?a=cr1_grp&rv=cr1&_af_eid=142278212&_af=event]], lightning talks by Pulse Energy and J. Bryant (UBC Stats) @ HootSuite
**[[HXD meetup: Designing a Career: From Clinician to App Developer|http://www.meetup.com/HXD-Vancouver/events/144625982/]]
!!References:
<<bibliography>>
11am Thursday
*Meeting schedule in the summer:
**06/07: TM
**06/14: TM+JM,
**''06/21: TM+JS''
**06/28: no meeting (TM, JM away)
**07/05: no meeting (TM, JM away)
**07/12: no meeting (JM, MB away)
**07/19: TM+JM
**07/26: TM+JM
**08/02: no meeting (MB, TM away)
**08/09: no meeting (MB, TM away)
**08/16: TM+JM
[[Overview]] Project
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[interview foci and questions|Overview Deployment Field Survey]]
*User #1: JW: Tulsa World reporter investigating emails re: police equipment purchases
**[[JW's story|http://tulsaworld.com/tpdtechemails]]
**[[project on DocumentCloud|http://www.documentcloud.org/public/search/projectid:%205269-tpd_emails]]
**JW sent missing overview lot file yesterday
*Other users' status? - keeping track in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**new user (MF, freelance) w/ sensitive data - barriers to research/interview?
2. [[An Insight-Based Evaluation of Overview]] with data journalism students
*to discuss w/ JS re: part-time instructor post this fall
*ethics for ~JSchool student project TBD when JS's teaching post / ability to conduct study confirmed
**amendment needed for Overview study w/ jSchool students
Overview development plan
*What's the timeline?
Using Overview (and other document mining tools, techniques, scripts):
* [[Visually exploring 13,000 music reviews with Overview|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]] - JS says this is the largest dataset used so far in Overview
*Overview w/ Vancouver city council meeting minutes (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]) (~2,500 pages)
*~2,000 news articles from various media outlets relating to Vancover's DTES (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]])
Overview misc:
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: http://lensingwikipedia.cs.sfu.ca/
**meeting Jul 25 4:30pm (following ~InfoVis)
*email thread w/ K. Melnick, [[VIVA VA challenge group|https://sites.google.com/site/challengeva/]] member re: Overview -gave a demo of Overview on Monday Jun 18, discussed Vancouver Sun collaboration
**also collaborating w/ Anoop
*met K. Rozendal, Vancouver Sun reporter covering Saul's visit, interested in Overview; also a UBC ~J-School student - interested in using Overview with Vancouver city council meeting minutes
>//"I think seeing your pre-processing strategies and the tools you used would be a nice element of a demo. At the moment, there's not much I could do with either the recipe or the final dish of docs, but maybe we can sit down later in the summer for a meeting where I can take a look. What sort of blocks on your time do you have in the next few weeks? Given the local and political nature of the content, the minutes would be the perfect raw material for showing some of us at the j-school a bit about visual analytics.//"
Upcoming / ongoing:
*SIGGRAPH ASIA review request, July 13 - sanity check to TM
*DRITW rejected from ~InfoVis - fast tracked to TVCG - next steps?
**to meet w/ TM, MS, SI (1 week1 from now after ~InfoVis revisions submitted for DSM, ~RelEx)
*~ToCHI journal paper on domestic interruptions with CT, JM
**TOCHI In The Wild special issue July 2 deadline
*vacation middle week of July (12-13, 16-17, 20) / beginning of August (July 27 - Aug 9)
!!References
<<bibliography>>
*Meetings in July: JM+TM: 07/19, ''TM+JS07/26''
[[Overview]] Project
1. [[Document mining in data journalism]] - post-deployment field data collection from users
*[[ongoing analysis presentation|https://dl.dropbox.com/u/6397998/docmining_analysis_07.19.pdf]]
*Other users' status? - keeping track in [[GDoc|https://docs.google.com/spreadsheet/ccc?key=0AnRg-4kycp0QdGs2OFlqYWRFRnVROGowODRSazh0MlE#gid=0]] / Dedoose
**new Finnish election questionnaire data user: TBD, possible visualization as product as well as tool
*Next steps: refining the [[interview foci and questions|Overview Deployment Field Survey]]
*ideas about personal information management in visualization, incremental tagging vs. one-shot en-masse tagging, structured vs. unstructured information
*user goals: smoking gun (read most/all) vs. major trends (read some)
**could reflect difference between document requests (ԐD) vs. document dumps (ɲaqi war logs)
**how to deal with weird / unexpected
*early 00's VA research on the visual display of search results / faceted browsing: much relies on incorrect assumption that users should be aware of the entire search space
*possible usage pattern: the U-shaped curve for document read times
*tags vs. flags (directive tags / action tags)
**how much cross-cutting of tags and flags? nesting tags?
**"show me everything that's not tagged" meta-tag (boolean)
*tagged document URL exporting
**evidence-for/against: hypothesis management / [[HunchWorks|http://www.unglobalpulse.org/technology/hunchworks]]
*web-based Overview Alpha in September at Online News Assoc, conference (not as tech-centric as NICAR)
**Document Cloud barrier less of a restriction than current Ruby/config/install barrier (eventually Document Cloud constraint will go away)
**developed in Scala
**new tree layout: collapse/expand nodes, pruning by node size, pruning by depth; node size proportional to number of documents in node; panning and zooming; only active tag colours nodes
***explicit (expand/collapse) vs. implicit (screen real estate) tree interaction (see M. McGuffin papers on big tree interaction)
**implications for logging
*Reflections:
**current data analysis process overkill for design purposes; with additional users, some interesting usage patterns might emerge; nevertheless, JS ecstatic to have rich user data
**future interviews: worth transcribing? worth doing log file analysis? perhaps only if usage pattern is suspected to differ substantially from existing users (decide on-the-fly)
2. [[An Insight-Based Evaluation of Overview]] with data journalism students
*study more likely to occur in Winter 2013 with Overview 0.1 Alpha
3. Other Overview-related "thingies":
*[[Anoop Sarkar|http://www.cs.sfu.ca/~anoop/]] SFU prof specializes in NLP: [[Lensing Wikipedia|http://lensingwikipedia.cs.sfu.ca/]]
*met with [[VIVA VA challenge group|https://sites.google.com/site/challengeva/]] Monday Jun 25: demo and Q+A
**interested, no one currently doing unstructured data analysis
**also collaborating w/ Anoop
*met K. Rozendal, Vancouver Sun reporter, interested in Overview; also a UBC ~J-School student - interested in using Overview with Vancouver city council meeting minutes
Using Overview (and other document mining tools, techniques, scripts):
* [[Visually exploring 13,000 music reviews with Overview|http://matthewbrehmer.net/2012/05/28/visually-exploring-13000-music-reviews-with-overview/]] - JS says this is the largest dataset used so far in Overview
*Overview w/ Vancouver city council meeting minutes (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]]) (~2,500 pages)
*~2,000 news articles from various media outlets relating to Vancover's DTES (from [[VIVA VA challenge group project page|https://sites.google.com/site/challengeva/projects]])
Recent and ongoing:
*reading:
**<<cite Budiu2009 bibliography:Bibliography>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Grimmer2011>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Marchionini2006>>: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
**<<cite Wagstaff2012>>: ML that matters - [[notes|Information Visualization Evaluation: Meta-Analysis]]
**<<cite Heer2007a>>: asynchronous collaboration and visualization: [[notes|Information Foraging, Sensemaking, Insight, & Serendipity]]
*reading gleicher write-ups on [[phd theses|http://pages.cs.wisc.edu/~gleicher/Web/Advice/PrelimsAndThesis]] and [[status reports|http://pages.cs.wisc.edu/~gleicher/Web/Advice/StatusReports]]
*vacation: weeks of Jul 30, Aug 6
*@TM: opportunities for studying unstructured text/document mining at Twitter?
**they are toolbuilders, have their own Vis team, see [[EuroCup 2012 Visualization|https://euro2012.twitter.com/]]
!!References
<<bibliography>>
!!Agenda
*Recent Overview UI changes; JS' new demo video
*JS' Overview think-aloud usability notes
*CHI paper on Overview tasks
*Overview + journalism student study
*~MoDisco
!!Notes
*CHI paper: a paper about what we have learned, about how we didn't initially know what the problem was; not a paper about Overview itself (CHI is not a forum for introducing / advertising Overview); draft to JS before Sept 4 when ready, draft to SI Sept 4, draft to ~InfoVis group Sept 11; reconcile typology analysis story with design chronology
**New story completed w/ Overview; JS to schedule interview Aug 19-21
**Relating Overview tasks to JS' high-level goals of transparency; citing the "does transparency work" literature; integrating the [[Text Analysis in Transparency 䡬k at Sunlight Labs|http://overview.ap.org/blog/2013/05/video-text-analysis-in-transparency/]] content: lessons learned, high-level tasks, metrics, goal of transparency
***data-driven low-hanging fruit vs too/technique-driven capacity building vs what-happens-if where impact=supply+demand+effectiveness (contributes to agency)
***transparency as deterrence vs attention vs understanding (secrets vs mysteries) vs influence mapping
***transparency grand challenge: illuminate for citizens how the decisions that affect them actually get made. which requires figuring that out. show them how to use their own influence.
**To view: JS' [[NewsU|https://twitter.com/newsu]] webinar [[Document Mining with Overview: A Digital Tools Tutorial|http://www.newsu.org/digital-tools-overview]]
**task typology on search: lookup / locate / browse / explore
***Overview user goals: finding the smoking gun (single or handful of docs) vs. finding the trend across all/many docs
***Overview as a reading tool, a way to systematically read all docs; produces a summarization; read all vs. skip most
**In case of Overview, Task typology not a design tool, we're using the typology retroactively to analyze user tasks
**retain barriers to use section (pre-paper slides 38-39), don't be apologetic
**Overview UI changes
***MOST/ALL keyword labels on nodes: these are sanity check on the clustering, serving the same purpose as the original MDS plot, a 2nd order measure of confidence / cluster quality; at high N, scatterplot is unreadable; relating to DRITW work
***JW used the MDS plot to look for orphans; one could imagine a "read/unread" meta-tag, or an "untagged" metatag/pseudotag
***max number of cluster children is 5, cluster width encodes size of cluster; otherwise reading singleton clusters becomes linear search
***discussing the interconnectedness of visual encoding, interaction, and task; matching tasks to the various versions of the tree, the MDS plot
**Why do people stare at little dots?
>''JS'': //I have asked myself this question too. My answer://
>
>//- clusters are very general structures//
>//- they're robust to transformations//
>//- 2D spatial clusters are easily visually perceptible//
>
>//You might also be interested in outliers or pattern matching or other types of general features, which your visualization system could support, but clusters are often interesting. Clusters are a robust structure, by which I mean relatively easy to preserve during various kinds of data transformations and visual encodings -- pretty much any continuous mapping will preserve clusters. And it's easy to spot the clusters in a 2D plot of some sort.//
>
>''TM'': //I agree that staring at dots is related to answering questions about clusters, and agree with your point about the generality/robustness/perceptibility of clusters. But then the next level down is OK, so what kinds of question about clusters are being asked? And yesterday I raised the issue of inspecting cluster *contents* (which you can do in an individual detail-on-demand way in many systems by clicking on points to see more details about what that point represents), vs inspecting cluster *quality* (which you can do directly by looking at their placement). And yup, finding outliers is another thing you can do with staring at dots - both notice their existence, and inspect their contents.//
>
>''JS'': //So, it's a pretty general starting point for a general data exploration primitive. I think I mean "explore" in the sense of your typology but not certain.//
>
>//I think you mentioned a recent paper asking this "why do people want to stare at dots" question?//
>
>''TM'': //That recent paper is DRITW, another one where it's a long road and the reviewers have not yet seen the light. It's currently under a third round of review. There's a significant chance that it will be rejected, given the hostility of the reviewers in the last round. It could be that if we don't have room to get deep into the stare-at-dots question in this CHI paper then it might have a place in a next draft of this one...//
>
>''JS'': //Re our Overview paper, I think maybe one of the good lessons here is that the same tool can be used for remarkably different analysis pipelines. It may not be possible for the visualization designer to see all of these uses before the system is in the field, both because they are not a domain expert and because the best tools change practice as they are deployed. Then the typology is a method for being able to see this.//
>
>//"reet finds its own uses for things"// - William Gibson, //Burning Chrome//
*JS' journalism student study:
**not to factor into CHI paper
**this serves a different purpose from that of the CHI paper, it is about what visualization can do for journalism, whereas the CHI paper is about visualization in itself
**is there a gold standard story? Does Overview allow for the different journalists to write the same story?
**given the same document set, will journalists in NY write a different story from those in BC, HK; gender / age / experience differences?
*SI's ~MoDisco research:
**discussing the ~MoDisco marketing problem; to split into 2.5 papers, showing IR alogrithms as DR algorithms
**~WikiLeaks 250k cables dataset easily available via torrent
**Trek dataset from NLP Linguistics Data Consortium
**~MD-SNE vs. ~Barnes-Hut tSNE
**Not all HD datasets are the same, there's a sparsity, but a structure to the sparsity
Meeting w/ Michael Sedlmair 10am re: HDD ethnography project
*What's been done so far?
*Future plans for the project
*My contributions / possible research angles
There likely won't be time to discuss my other ongoing work, so it's all documented here:
*Project idea: text visualization for journalism
**[[3 Difficult Document-Mining Problems that Overview Wants to Solve|http://www.pbs.org/idealab/2011/10/3-difficult-document-mining-problems-that-overview-wants-to-solve297.html]] by J. Stray
**[[TheOverview Project|http://overview.ap.org/]] - they want to build [[processing|http://processing.org]] for text visualization. Lightweight, no extensive programming knowledge required. Casual use by journalists with large datasets
**Some possible field research questions in here:
>"What are the most painful things that journalists must do with document sets? What are the things they髥 to do but can嵐y? What problems exist, and which should we focus on solving? The only way to answer these questions is to talk to users."
*Project idea: UBC Text summarization research (R. Ng, G. Carenini) (JM's suggestion)
Meanwhile, my ongoing [[Literature Review]] / what I've been [[Reading|References]] last week/this week:
*[[References: Read]]:
**[[Information Visualization Evaluation: Meta-Analysis|]]
***<<cite Kang2010 bibliography:Bibliography>> - BELIV '10 position paper
***<<cite Elmqvist2010>> - BELIV '10 position paper
***<<cite Chen2000a>> - Chen & Czerwinski early meta-review, not much value added now
***<<cite Thomas2005>> - PNNL's //Illuminating the Path// ch. 6
**[[Information Visualization Evaluation: Qualitative Methods]]
***<<cite Gonzalez2003a>> - 2nd paper about long-term workplace study w/ negative results
***<<cite Valiati2006>> - BELIV '06 task taxonomy paper
***<<cite Perer2009>> - conducting MILC studies
***<<cite Zuk2006>> - BELIV '06: using and comparing 3 sets of ~InfoVis heuristics
***<<cite Faisal2008>> - BELIV '08 - grounded evaluation
***<<cite Tory2008>> - BELIV '08 - grounded evaluation
***<<cite Valiati2008>> - BELIV '08 conducting a MILC
***<<cite Tory2005>> - expert / heuristic reviews
***<<cite Lloyd2011>> - ~VisWeek '11 ~GeoVisualization paper (Michael's recommendation) - long-term case study
**[[Information Visualization Evaluation: Quantitative Methods]]
***<<cite Sutcliffe2000>> - quant. study w/ negative / misleading results
***<<cite Chang2010>> - BELIV '10 learning-based evaluation (I like this one)
**[[Information Foraging and Sensemaking]]
***<<cite Pirolli2009 bibliography:Bibliography>> - ch. 1 - explaining the theory
***<<cite Pirolli2009 bibliography:Bibliography>> - ch. 9 - design heuristics for the web
***<<cite Thomas2005>> - PNNL's //Illuminating the Path// ch. 2
***<<cite Yi2008>> - BELIV '08: "how and when is insight gained?" not "what is insight?"
**[[Evaluation in HCI: Meta-Analysis]]
***<<cite Greenberg2008>> - usability evaluation considered harmful (CHI 2008) (MUX discussion #1)
***<<cite Crabtree2009>> - ethnography considered harmful (CHI 2009)
**[[Research through Design]]: <<cite Zimmerman2010>> (MUX discussion #2)
*[[References: To Read (Priority)]]
**papers & book chapters on deck / ongoing
*[[References: To Read]] (lower priority)
**roughly 100 papers or book chapters
**now thematically categorized
**I've been broadening the scope to other HCI communities (CSCW, Creativity & Cognition, etc.)
**~VisWeek: send relevant papers my way; I will be browsing the proceedings too.
Recent events:
*~VisWeek - I will be browsing the proceedings for ~InfoVis group meeting
*Distinguished Lecture on Design Engineering: //Dancing with Ambiguity: Embracing the Tension between Innovation and Decision-making in the Design Process//. Larry Leifer, Dept. of Mechanical Engineering, Stanford University [[http://cdr.stanford.edu]]
*Grad Student Workshops and Events, [[Graduate Pathways to Success Program|http://www.grad.ubc.ca/current-students/gps-graduate-pathways-success/gps-workshops-events]]
**[[Academic Researcher to Commercial Writer: Workshop Notes]]
**Project management seminar(s): Oct 25, early Dec
**Technical writing: early Dec
Upcoming:
*holiday travel departing Wed Dec 14, working remotely Dec 15-16, 19-23, returning Wed Jan 4
*travel February 2012 TBD (approx 1 week)
!References
<<bibliography>>
!![Ahlberg1994] - Visual information seeking
<<cite Ahlberg1994 bibliography:Bibliography-TaskTaxonomy>>
!!!!Comments & Questions
*
!![Ahn2014] - task taxonomy for network evolution analysis
<<cite Ahn2014 bibliography>>
!!!!Comments & Questions
*
!![Aigner2007] - Visualizing time-oriented data
<<cite Aigner2007>> is later elaborated on in the <<cite Aigner2011>> textbook.
!!!!Comments & Questions
*cited by R2:
>//A weakness of the related work is that the "what" part is greatly underrepresented. This is intentional, but I disagree about the decision. There should be a least examplary publications about this, for example starting with the task by data type taxonomy by Shneiderman which is already mentions as [56] in Section 2.3, but does only provide a minor "how" answer in Chapter 3. From there, examples for each basic datatype are possible. For time, this could be// <<cite Aigner2007>>
!![Aigner2011] - role and value of interaction in VA
<<cite Aigner2011>>
!!!!Comments & Questions
*
!![Aigner2011a] - //Visualization of ~Time-Oriented Data// (text)
<<cite Aigner2011a>>'s textbook breaks down the visualization of time-oriented data into 3 questions: //what// is being visualized (data and time (and sometimes space)), //why// it is visualized (user tasks), and //how// the visualization is designed (the visualization pipeline). (see [[notes|Information Visualization: Techniques]].). The characteristics of time are: scale, scope, arrangement, viewpoints (ordered vs, branching time), granularities, time primitives, and determinancy.
Ch. 4 summarizes several relevant task models, mentioning <<cite Winckler2004>> and its use of ConcurTaskTree (CTT, <<cite Paterno2002>>), a task modelling language that breaks tasks down into abstract, user, application, and interactive tasks). They also summarize the work of <<cite MacEachren1995>> (identification vs. localization) and <<cite Andrienko2006>> w.r.t. tasks. The latter they cast as a taxonomy, as described below in the discussion of <<cite Andrienko2006>>.
Ch. 5 on interaction in visualizations of time-oriented data, describing the intents of interaction using the definitions of <<cite Yi2007>>: //select, explore, reconfigure, encode, abstract/elaborate, filter, connect, undo/redo//. Also discusses <<cite Spence2007>>'s notions of //stepped// vs. //continuous// interaction, also discussed <<cite Norman1988>>'s stages of action, as well as the model-view-controller pattern. (see [[notes|Information Visualization: Techniques]].)
In Ch. 6 (analytical support), they discuss higher-level analysis tasks to which visualization of time-oriented data contributes to. These are:
*''classification''
*''clustering''
*''search and retrieval''
*''pattern discovery''
*''prediction''
These fall under the higher-level goal of ''data abstraction'', to reduce workload and keep perpceptual effort low, discarding irrelevant data automatically and focusing in on abstractions: classifications, clusterings, patterns. The chapter goes on to discuss clustering, vertical and horizontal temporal data abstraction*, and PCA (for pattern discovery) in greater detail. For the latter, they provide guidance for performing PCA with time-oriented data, in that time is usually excluded from the PCA process and reintroduced when visualizing the principle components.
*''vertical data abstraction'' refers to instants/events in time, including co-occurrences; this may be qualitative abstraction (event labeling), generalization abstraction (event categorization), definitional abstraction (event categorization from multiple typologies/taxonomies)
*''horizontal data abstraction'' refers to intervals / spans in time, including merge/state abstraction (sequence labeling), persistance abstraction (labeling before and after events/intervals), trend abstraction (labeling a gradient or rate), periodic abstraction (labeling a repeating pattern)
!!!!Comments & Questions
*an extension of <<cite Aigner2007>>
*vertical and horizontal temporal data abstraction = fancy name for classification? for automatic chart annotation? or use of aggregated glyph representations?
!![Amar2004] - Analytical Gaps (~High-Level Task Taxonomy)
<<cite Amar2004>> and <<cite Amar2005a>> discuss [[Analytic Gaps]], barriers to higher-level analytic tasks (decision making, learning) when using [[Information Visualization]] tools. These gaps can be differentiated between a ''rationale gap'', between perceiving a relationship and appreciating the utility and confidence of the relationship, and a ''worldview gap'', between what is shown to a user and what is hidden from the user, the amount of information needed to make a conclusion and decision based on the data.
The authors propose an systematic valuation method akin to [[Heuristic Evaluation]] by expert evaluators, identifying analytic gaps to assess whether the tool supports higher-level cognitive analytic tasks.
Systems designed and evaluated with consideration of [[Analytic Gaps]] will focus on ''analytic primacy'', supporting high-level tasks, rather than ''representational primacy'', which focuses on low-level tasks. However, a balance must be struck such that the tool is not a //black box//, in that the underlying data can still be viewed and interpreted: user control must be provided.
A taxonomy of analytic knowledge tasks (decision making under uncertainty, learning a domain) is provided for conducting the evaluation:
*''Rationale''-based tasks
**''expose uncertainty'' (measures and aggregations, effect of uncertainty on outcomes)
**''concretize relationships'' (what comprises the representation of a relationship and its outcomes)
**''formulate cause and effect'' (clarify possible courses of causation)
*''Worldview''-based tasks
**''determine domain parameters'' (provide transfer/acquisition of knowledge or metadata about domain parameters within a data set)
**''multivariate explanation'' (allow discovery of correlation and constraint)
**''confirm hypotheses'' (allow formulation and verification of hypotheses)
The authors conclude the paper by applying these tasks in both a design and evaluation setting.
!!!!Comments & Questions
*journal version: Amar, R. and Stasko, J. (2005). Knowledge precepts for design and evaluation of information visualizations. IEEE Trans. Visualization and Computer Graphics (TVCG), 11(4): 432
*re: section 1.1: Decision makers rely more on the the macro-level for making decisions. Breadth rather than depth? This seems at odds with the idea that breadth-level insight is shallow and not as valuable as depth-level insight (<<cite Saraiya2004>>).
*re: section 1.2: Does each new scientific domain require a new visualization?
*re: section 1.3: claim that most systems do not represent ''uncertainty'', ''cause and effect'' particularly well.
*No mention of whether evaluation of analytic gaps requires domain expertise, other logistical constraints.
*If another dimension of the taxonomy is decision-making vs. learning - where do the knowledge tasks fall?
*<<cite Saraiya2004>>'s notion of ''insight'' is related to the high-level knowledge tasks referred to here. <<cite Amar2004>> use the taxonomy of knowledge tasks to explicitly define what insight is needed for.
**The former's [[Insight-Based Evaluation]], with a domain expert, can be analyzed quantitatively, but without regard to specific knowledge tasks.
**<<cite Amar2004>>'s taxonomy does not place different values on knowledge tasks and thus evaluations are qualitative and results must be analyzed on a case-by-case basis (as some domains will place different values on the different knowledge tasks).
**Could a joint evaluation framework be devised that combines the taxonomy of knowledge tasks with [[Insight-Based Evaluation]]?
*There exists a later journal paper, however it does not contribute enough additional material to make it worth reading.
!![Amar2005] - Low Level Analytical Components
<<cite Amar2005>> aimed to characterize low-level visual analysis tasks. They recruited students to generate lists of questions, given data sets and visualization tools. The authors used affinity diagramming to reduce these questions to a set of 10 low-level tasks:
**''retrieve'': find attribute values for cases
**''filter'': find data cases satisfying a condition on attribute value(s)
**''compute derived variable'': compute an aggregate value for a set of data cases
**''find extremum'': find extreme values for attribute(s)
**''sort'': sort cases by attribute(s)
**''determine range'': fid range of values for attribute(s)
**''characterize distribution'': determine distribution of quant. attribute for a set of cases
**''find anomalies'': outlier detection, unexpected values
**''cluster'': find similar cases
**''correlate'': do attributes have a relationship?
These tasks can be chained/compounded, some appearing as primitives in others.
The authors characterize other low-level tasks that are not visual analysis tasks, being mathematical or cognitive operations:
**''compare'': is value x = value y?
**''verify filtering criteria'': a mathematical operation
They also omit high-level tasks and uncertainty decisions (exploratory data analysis), some of which are accounted for in their high-level taxonomy (<<cite Amar2004>>):
**''decision making under uncertainty'': often involve a value judgment beyond low-level visual primitives
**''learning a domain'': could involve repeated ''correlate'' and ''characterize distribution'' primitives
**''trend identification'': similar to the learning case
**''prediction'': (involving ML techniques into visualization workflow)
!!!!Comments & Questions
*CPSC 533C comment (09/16/09): Can exploratory analysis (hypothesis forming / confirmation) always be reduced into the deductive primitive analysis tasks mentioned by Amar et al? What additional low-level tasks might emerge in exploratory visualization tools, such as Improvise?
*their methodology (student participants, lab setting, short time span) cannot account for longitudinal exploratory data analysis, hypothesis generation and validation (trend identification, learning a domain)
*in <<cite Roth2012>> meta-taxonomy
!![Andre2009] - On Serendipitous Discovery
<<cite Andre2009>>'s Creativity and Cognition conference paper reviews the definitions of serendipity, historical accounts of ''serendipitous insight'' and ''discovery'', and how it has been interpreted in the Computing Sciences. It offers a meta-review of systems built to support serendipity, however the paper argues that these systems only account for ''one part of serendipity'': the ''chance discovery of something unexpected'', or ''something sought after in an unexpected location'' (the cause). Many systems do not account for the ''second aspect of serendipity (the effect)'', the //sagacity// or insight to ''acknowledge an unexpected connection with earlier knowledge and expertise'', and the will to ''act upon these connections'', by ''reinforcing an existing problem or solution'', ''rejecting or confirming ideas'', or ''starting a new research direction''. There exists many existing systems that filter or suggest potentially relevant or interesting content to a user, akin to a recommender system (albeit recommender systems often rely popular content, as opposed to personalized content). These systems tend to be peripheral/background applications and occurrences of serendipitous discovery are rare, however users tend to be delighted when this happens. More often these systems are distracting or overwhelming.
The paper concludes with high-level design recommendations for providing opportunities to support this second aspect of serendipity, that ability to act on unexpected but useful information by drawing connections through analogy, inversion, or successful error. Systems would therefore have to prime users with familiar content in order to draw connections with new unexpected content, by exploiting a shared metalanguage or semantics across disciplines. Such a shared metalanguage would allow for serendipity-hunting agents to be built. Coupled with personalized content presentation (background/peripheral reporting, i.e. feed-readers) and a detailed understanding of a user's expertise (via life logging, heterogeneous sources of user information), this pairing could facilitate the second aspect of serendipitous insight. Integrating retroactive answering of search queries, novel presentation of search query results, and activities involving creativity, play, and aleatoricism may also promote better chance encounters (the first step of serendipity).
The authors also remark that the study of serendipity and insight is difficult due to their inherent rarity, particularly in a controlled setting. In naturalistic interview, diary, or search log studies, it is often difficult to identify specific instances of serendipitous discovery and insight. Nevertheless, they acknowledge the value in multiple approaches to studying this complex and rare phenomena across domains.
!!!!Comments & Questions
*The intro/motivation goes at great length to separate the two defining aspects of serendipity, however the structure of the rest of the paper dances back and forth between the two, leaving me questioning as to how much of a contribution they have made in terms of design recommendations for the second aspect of serendipity-supporting systems
*Design recommendations are dangerously hand-wavy, treading into Malcolm Gladwell territory (he is cited in the paper), history of science lessons
*Shared metalanguage across disciplines - I would need to familiarize myself w/ computational linguistics / semantic web research to see how far away we are from achieving this dream
*Differences between recommender systems and serendipity agents is still hazy despite their efforts to disentangle the two
*Designing personalized systems is the key to insight?
*Still a large reliance on the user to "have a prepared / open / questioning mind". Motivation and engagement will interact with the amount of priming needed to make serendipitous connections
*Glad they acknowledged the difficulty of measuring/quantifying insight and serendipitous discovery, even in naturalistic studies
*Hoping they would have expanded on the role of creativity and play in serendipitous discovery, perhaps with historical examples.
*Privacy concerns and resistance to "life logging", information-seeking tasks still distributed across many (potentially) unconnected devices and platforms, some of which not digital.
!![Andrienko2003] - spatio-temporal visualization
<<cite Andrienko2003>>: ''objective-based taxonomy'': identify, compare; ''operand-based taxonomy'': space, time, objects
*task typology includes 3 dimensions: cognitive operation (operator), search target, and search level
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
!![Andrienko2006] - //Exploratory Analysis of Spatial and Temporal Data// (text)
<<cite Andrienko2006>> introduce a ''typology'' of EDA tasks in which they distinguish between elementary and synoptic tasks, taxonofied by <<cite Aigner2011>> as follows:
*''elementary tasks'' (on values): individual elements (low-level, serving a marginal role)
**''direct lookup'' (identification)
**''inverse lookup'' (localization)
**''direct comparison'' (identification - interrelate characteristics)
**''inverse comparison'' (localization - interrelate references)
**''relation seeking'': implies a search for occurrences of specified relations between characteristics or between referents.*
*''synoptic tasks'' (on sets): whole set of elements or subsets (primary role: most tasks are synoptic)
**''descriptive tasks'':
***''direct lookup'' (pattern defintion, identification)
***''inverse lookup'' (pattern search, localization)
***''direct pattern comparison''
***''inverse pattern comparison''
***''relation seeking''
**''connectional tasks'': "behaviour is a generalisation of such notions as distributions, variations, and trends; for example, the variation of the proportions"
***''identify heterogeneous behaviour''
***''identify homogeneous behaviour''
The Andrienkos define a task as follows:
>//A task is viewed as consisting of two parts: the target, i.e. what information needs to be obtained, and the constraints, i.e. what conditions this information needs to fulfil.//
The distinction between elementary and synoptic is inspired by Bertin's systematic distinction of //question types//, typical questions that need be answered by means of data analysis involved knowns and unknowns at several levels of reading: ''elementary, intermediate, overall''. This mirrors the structure of the data, and is free from bias specific to particular tools or analysis methods. They extend Bertin's framework by including comparison tasks in addition to lookup tasks.
They use a formal quasi-algebraic notation to express tasks in terms of referents (points) and characteristics (attributes).
>//The rationale for using the formal notation is to build a well-grounded, distinct, consistent, complete task typology, which means://
>1. It must be clear where each task type comes from and why it is introduced.
>2. It must be clear how one task type differs from another.
>3. A common approach is used to define all task types.
>4. There is a way to confirm that all potential data analysis tasks have been taken into account.
Targets for synoptic tasks are types of patterns (behaviour), or a particular configuration of characteristics for a reference set; they include:
*association: patterns based on similar values
*differentiation: patterns based on differing values
*arrangement: patterns of arrangement in time and space
*distribution summary: patterns of distributions and outliers
They do not aspire to creating a full classification of patterns. They describe these four in detail and discuss implications for multidimensional patterns. They also discuss how the same elementary tasks of lookup, comparison, and relation seeking can be applied to sets, thereby becoming synoptic tasks.
Connection discovery tasks are about explaining behaviour whereas synoptic tasks merely summarize it (descriptive). This regfers to finding causal/logical/structural links, correlations, or influences between phenomena. It is not an extension of behaviour comparison, but a special task category of behaviour characterization that "deals with detecting signals of possible inherent connections and interactions". Whether the connectionist task pertains to homogenous / heterogeneous behaviour depends on the type of phenomena, the latter referring to complex phenomena.
According to <<cite Aigner2011a>>:
>//In contrast to descriptive tasks, connectional tasks establish connections between at least two sets, taking into account the relational behavior of two or more variables. Depending on the set of underlying references other variables are considered over the same set or over different sets of references hoogeneous and heterogeneous behavior tasks are distinguished.// (p. 75)
They claim that their task typology is complete, nor is it specific to spatiotemporal data. But is it usable?:
>//We hope that, despite the use of some formal notation, our scheme is still understandable and does not require a solid mathematical background.// (p. 153)
<<cite Aigner2011a>> see their notation as necessary for automating the design process
On whether it is important or necessary:
>//Some of our colleagues believe that having a defined task is not (or not always) necessary in information visualisation. Others are convinced that tasks always exist, explicitly or implicitly, even when an explorer seems to look for data.// (p. 148)
They address <<cite Shneiderman1996>>'s mantra and see that their typology is consistent with this, in that analysts start with synoptic tasks (overview first) and "cut" the behaviour into smaller pieces, investigating some of these in turn (zoom and filter, details on demand). Their formal specification allows for bottom-up analysis just as their typology categories allow for top-down analysis of tasks. Crosscutting this is the distinction between analysis and synthesis.
On other task taxonomies:
>//Therefore, we are not going to undertake a detailed analysis of every existing taxonomy in order to reveal its weaknesses in comparison with our superb task typology.// (p. 152)
They discuss/compare against <<cite MacEachren1995>> (existence of data element, temporal location, time interval, temporal pattern, rate of change, sequence, synchronization), <<cite Peuquet1994>> (what / where when), <<cite Shneiderman1996>>'s mantra, <<cite Zhou1998>>, <<cite Wehrend1990>>, <<cite Blok2000>> (identification and comparison, cited in <<cite Roth2012>>), Gahegan and O'Brien (1997: exploration, search, comparison), <<cite Casner1991>>, Qian (1997, data-centric), Jung (1995), <<cite Robertson1991>>, <<cite Roth1990>>, <<cite Knapp1995>>, Kraak (1997), Albrecht (1995, 1997), Klir's mathematical formulation of Bertin's questions (1985), and data mining tasks (Fayyyad 1996, Miller 2001): segmantation (clustering, classification), dependency analysis, deviation and outlier analysis, trend detection, generalization and characterization.
In summary, their typology of tasks is to be used for determining what tasks a particular tool could support (descriptive power), planning to support tasks when specifying/designing a new tool (generative power), and having data, determining which tools and tasks to perform and in what sequence/combination
!!!!Comments & Questions
*a difficult read, explained much more concisely by <<cite Aigner2011a>>
*cited by <<cite Aigner2011a>>, a list of spatiotemporal visualization tasks, arranged into a taxonomy, they see ''relation seeking'' as an extension of ''comparison''
*on previous typologies of tasks (they explicitly don't use the term taxonomy):
>//many other task typologies, which simply enumerate some tasks without providing sufficient background for selecting those particular tasks and without any judgement concerning the completeness of the suggested list.//
!![Andrienko2007] - Geovisual analytics for spatial decision support
<<cite Andrienko2007>>
!!!!Comments & Questions
*
!![Attfield2010] - Sensemaking in visual analytics: Processes and challenges
<<cite Attfield2010>>
!!!!Comments & Questions
*
!![Bailey1994] - taxonomies and typologies: intro to classification
<<cite Bailey1994>> on taxonomies vs. typologies:
>//The basic difference, then, is that a ''typology'' is conceptual while a ''taxonomy'' is empirical. Exceptions to this generally involve the subsequent identification of empirical cases for conceptual typologies, but not the conceptualization of taxonomies. (p. 6)//
Taxonomies are the classification of empirical entities, used in biosciences, whereas typologies are social science territory. Taxonomies are often but not always hierarchical.
!!!!Comments & Questions
*
!![Bates1989] - browsing and berrypicking
<<cite Bates1989>>
!!!!Comments & Questions
*
!![Bautista2006] - task-based framework for preferential choice visualizations
<<cite Bautista2006>>
!!!!Comments & Questions
*
!![Bavoil2005] - ~VisTrails
<<cite Bavoil2005>>
!!!!Comments & Questions
*
!![~Beaudouin-Lafon2004] - Interaction Models for Descriptive, Evaluative, Generative Power
<<cite Beaudouin-Lafon2004>>'s AVI position paper about the power of thinking about interactions rather than interfaces. Discusses interaction paradigms: computer-as-tool, computer-as-partner, computer-as-medium, as well as interaction models: instrumental interaction (describing degrees of indirection, integration, conformance), situated interaction (describing context of use, flexibility of use, reinterpretability, resilience, scalability), and interaction as a sensory-motor phenomenon. Models can be evaluated on their:
*''descriptive power'': ability to describe a range of existing interfaces
*''evaluative power'': ability to help assess multiple design alternatives
*''generative power'': ability to help designers create new designs
>//High level-models tend to have good descriptive power but poor evaluative and generative power. Low-level models tend to have poor descriptive and evaluative power, but higher generative power. A good interaction model must strike a balance between generality (for descriptive power) concreteness (for evaluative power), and openness (for generative power).//
!!!!Comments & Questions
*Well-written position paper, powers of interaction model can apply well to VA/~InfoVis (models of analysis)
!![Bederson2003] - theories for understanding information visualization
<<cite Bederson2003>> on theories guide design by being //descriptive, explanatory, predictive, prescriptive//, and //generative//:
*"//describe objects and actions in a consistent and clear manner to enable cooperation// - clarifying terminology about objects and actions , key concepts, variables, guiding further inquiry and education, encourage commercial adoption
*//explain processes to support education and training//
*//predict performance in normal and novel situations so as to increase the chances of success// - accelerate development of effective ~InfoVis
*//prescribe guidelines, recommend best practices, caution about dangers// - development of improved interfaces
*//generate novel ideas to improve research and practice//" - guide innovation
Guiding principles and taxonomies are seen as worthwhile contributions to the study of the complexities of disciplines and human behaviour.
!!!!Comments & Questions
*cited by <<cite Liu2008>>
*similar to <<cite Beaudouin-Lafon2004>>'s generative, descriptive, and evaluative power
!![Bertini2010] - integration of automatic data analysis
<<cite Bertini2010>>
!!!!Comments & Questions
*
!![Beveridge1958] - art of scientific investigation (RR)
<<cite Beveridge1958>>
!!!!Comments & Questions
*
!![Blok2000] - geospatial EDA
<<cite Blok2000>>
!!!!Comments & Questions
*cited by <<cite Andrienko2006>>, <<cite roth2012>>
!![Bhatt2008] - Organizing knowledge in the knowledge development cycle
<<cite Bhatt2008>>
!!!!Comments & Questions
*
!![Brehmer2013] - multi-level typology of abstract visualization tasks (~InfoVis '13')
<<cite Brehmer2013>> describes a typology of abstract visualization tasks organized around three questions: //why// is the task performed, //what// are the task's inputs and outputs, and //how// is the task supported in terms of //methods// or //idioms// (families of interaction and visual encoding techniques). describing tasks using these three questions allows one to describe entire sequences of interdependent tasks with linked outputs and inputs:
*''why'':
**''consume'': //discover// (generate / refine / verify hypotheses), //present//, //enjoy//
***''search'': //lookup// (identity and location of target known), //locate// (only identity known), //browse// (only location known), //explore// (neither is known)
***''query'': //identify// (one target), //compare// (multiple targets), //summarize// (all targets)
**''produce''
*''what'': //inputs// and //outputs//
*''how'':
**''encode''
**''manipulate'': //select//, //navigate//, //arrange//, //change//, //aggregate//, //filter//
**''introduce'': //import//, //record//, //derive//, //annotate//
!!!!Comments & Questions
*
!![Buja/Cook/Swayne1996] - Interactive high-dimensional data visualization
<<cite Buja1996>>: ''operator-based taxonomy'': focusing, linking, arranging views
*Focusing (choice of [projection, aspect ratio, zoom, pan], choice of [variable, order, scale, scale- aspect ratio, animation, and 3-D rotation]), linking (brushing as conditioning / sectioning / database query), and arranging views (scatter plot matrix and conditional plot)
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
*in <<cite Yi2007>> RW
!![Buxton1986] - Chunking and phrasing and the design of human-computer dialogues [JM ref]
<<cite Buxton1986>> writes about distinct actions and compound tasks seen as single tasks dependent on input modality, also a perception associated with expertise:
>//Experts and novices differ in the coarseness of granularity with which they view the constituent elements of a particular problem or task. Novices are attentive to low-level details. with experts, these low-level details can be performed automatically. Hence, the size of the chunks of the problem to which they are attentive are much larger.//
>
>//One of our main arguments is that we can use tension and closure to develop a phrase structure to our human-computer dialogues which reinforces the chunking that we are trying to establish.//
The article explores interaction techniques designed around chunking single actions into compound actions. Where the ''pragmatics'' (device, kinaesthetic components) of an interface collapses non-terminal productions, we can treat these as virtual terminal nodes in a task decomposition, design prescriptively around these chunked tasks and facilitate the development of expert skills.
Foley, Wallace, and Chan (1984 IEEE CG&A)'s characterization of tasks from the user perspective reveals compound tasks seen as single tasks:
*''select'' an item in 1d, 2d, 3d
*''position'' an item in 1d, 2d, 3d (actually a compound of 2 ''quantify'' tasks)
*''orient'': rotate an item in 1d, 2d, 3d
*''path'': specify a path
*''quantify'': specify numerical value
*''text'': enter text
!!!!Comments & Questions
*In the //HCI: Toward the Year 2000// Anthology
*See Reisner (1981 IEEE Trans. Software Eng.)'s heuristics for the grammar of interaction language (for assessing system learnability, proneness to error): number of productions, number of terminals, length of productions (note: similarity to GOMS?)
*''pragmatics'' used in a subtly different way than Distributed Cognition folks (pragmatic = goal-directed)
!![Bystrom1995] - task complexity and information seeking - LIBR
<<cite Bystrom1995>>
!!!!Comments & Questions
*
!![Card/Mackinlay1997] - The structure of the information visualization design space
<<cite Card1997>> present a grammar / language for describing a mapping between abstract data types and common visualization types (data and presentation) (circa 1997). Of note, they distinguish between D (data) and D'(derived/transformed data), trasnformed by means of a transformation function F, which could take the form of ''filtering'', ''sorting'', ''MDS'', ''interactive input''), or ''unspecified function''). They futher distinguish between ''controlled'' and ''automatic processing'', the former referring to text features and the latter referring to marks - their retinal encodings and positions. Interaction is defined as ''view transformation'' and ''widget interaction'' (sliders, buttons).
They code several types of interfaces according to their scheme: scientific visualizations (Ozone concentration), GIS (Profit landscape), multi-dimensional plots (~FilmFinder, Worlds within Worlds), multi-dimensional tables (~TableLens), Information landscapes and spaces (NYSE Visible Decisions), Node and Link (internet traffic), Trees (Hyperbolic browser, tree-map, cone tree), text (~SeeSoft, ~ThemeScapes).
They don't directly approach the issue of tasks but instead refer in the summary to ''knowledge crystalization'' (<<cite Card1999>>), as well as ''information foraging'' and ''sensemaking'' (<<cite Pirolli2005>>). They acknowledge that combining this notation work with tasks would lead to an overly-complex notation.
!!!!Comments & Questions
*Users need to unmap the presentation to get at the data. Do users also need to unmap interactions to tasks?
!![Card/Mackinlay/Shneiderman1999] - Knowledge crystallization task [p.10-12]
<<cite Card1999>>'s ''knowledge crystallization'' task is a high-level hierarchy of tasks that occurs in a cycle not unlike <<cite Pirolli2005>>'s sensemaking loop. These tasks have the following characteristics: ill-defined problem solving, a need to communicate or act upon results, large amounts of heterogenous information, a well-defined goal (communication, decision-making) requiring insight. The stages are as follows, with information visualization supporting the sub-tasks:
*''information foraging'' (<<cite Pirolli2005>>): overview, zoom and filter, details-on-demand (<<cite Shneiderman1996>>), browse, search query
*''search for schema/representation'': reorder, cluster, class, average (derive new data), promote, detect pattern, abstract
*''instantiate schema w/ data'': reduce residue not fitting schema, improve schema
*''problem-solve to trade-off features'': read fact | comparison | pattern (Bertin 1977/1981), manipulate, create, delete
*''search for new schema that reduces the problem''
*''package the patterns found in some output'': extract, compose
The high-level goal is insight, which means ''finding a schema/representation'', an abstraction, if one already exists, this is ''information retrieval'', otherwise a schema will have to be generated through reduction and abstraction, a compact description.
!!!!Comments & Questions
*Stages connect to all other stages
*mix of high-level interaction and high-level goals; independent of data type
!![Carter1986] - A taxonomy of user-oriented functions
<<cite Carter1986>>
!!!!Comments & Questions
*
!![Case2008] - Models of information behaviour - LIBR
!!!ch. 6: Models of information behaviour
<<cite Case2008>> surveys models of information seeking, including two models of <<cite Wilson1999>>, as well as models of Krikelas (1983), Johnson (1997), and Leckie et al. (1996). The models are intended to generalize over a wide population, considering various needs and sources (other individuals, forms of media).
<<cite Wilson1999>>'s first model centres around the need of an information user, where the need elicits information seeking behaviour, which in turn involves demands on information systems and other sources, resulting in success (information found) or failure (information not found). In success cases, information is used, and the satisfaction or non-satisfaction of its efficacy feedbacks onto the information need. Meanwhile, information use can also involve transfer and exchange with other individuals. The model doesn't link failure cases back into needs, nor does it treat other individuals as sources.
<<cite Wilson1999>>'s second model places more emphasis on the information seeker's context and intervening variables (demographics, psychological state, motivation). It also discerns between passive attention (being exposed to information), passive search, active search, and ongoing search.
The Krikelas (1983) model revolves around uncertainty and the need to reduce that uncertainty. It distinguishes between //information gathering// in response to deferred needs and //information giving// in response to immediate needs. The former involves remembering information or making it available by storing it where it can be found. The latter involves selecting a source, choosing between internal (memory, personal files, direct observation), and external (interpersonal contact, recorded literature) sources. The model is outdated in that personal files and recorded literature are difficult to delineate. It doesn't take into account user characteristics.
The Johnson (1997) model involves several antecedents: background factors of demographics and direct experience with a topic of interest along with personal relevance factors (beliefs, salience - a perception that information is both relevant and applicable). These in turn affect the expected utility of information, which will dictate the actions that are subsequently taken, such as selecting information channels, distinguishing between active and passive information acquisition. Feedback loops occur between actions and antecedents.
Finally, the Leckie et al. (1996) model targets work- and task-related information seeking of professionals, and is thereby less general. These work tasks affect information needs and in turn the awareness of information and its sources. The model is a feedback loop between outcomes and needs.
!!!!Comments & Questions
*from JD's LIBR 553 2012 W syllabus
*Emphasis on general population, casual information seeking and non-work-related motivators.
*No emphasis on interaction with any specific form of media
*A shallow comparison of the models; doesn't offer any operational/actionable guidance, nor does it state how these models can be used. These models are descriptive rather than prescriptive or actionable.
*No concrete relation of these models to tasks. But they are related to the "why" question of data/information analysis, however it is too general to apply.
!!!ch. 7: Theories, perspectives, paradigms
This chapter relates to mid-level and high-level theories, as well as over-arching paradigms and theoretical perspectives relating to information behaviour. It attempts to distinguish between these related terms, covering some familiar ground (see [[EPSE 595]] notes), the subsuming structure of epistemology/ontology, theoretical perspective/paradigm, and mid-to-high-level theory or model: these being a "interrelated set of definitions, axioms, and propositions". It also makes references to <<cite Kuhn1962>>, describing how he uses the term "paradigm" in 21 different ways, often conflating the term with "theory". Case uses the terms "paradigm", "perspective", and "tradition" interchangeably, which subsume grand/formal theories, which in turn subsume middle-range or "grounded theories" and their observations. Case touches upon types of (constuctivist) research, referring to them as "critical" and "administrative" research traditions (as opposed to interpretive (phenomenological / symbolic interactionist) and critical). Case points out a need for middle-level "grounded theories, at a higher level than a testable hypothesis, but deal with limited settings, remain close to the level of observable phenomena, and offer the potential for aggregating findings".
Case reviews several of the major theories and their sources relating to information seeking behaviour, these emanating from psychology, sociology, communications, management and business, consumer research, economics, and linguistics. No overarching or comprehensive theory of information behaviour currently exists, though Case believes that theories emerging from the humanities, rather than psychology, should be embraced to a larger extent. Theories reviewed include:
*''Zipf's Principle of Least Effort'': individuals try to minimize the work they have to do, and media use and placement follows the 80-20 rule (following a harmonic distribution). Pragmatic yet suboptimal, related to a cost-benefit paradigm.
*''Uses and Gratifications'': Needs generate expectations, which require gratification through usage: "we use X because X gratifies us". Individualistic, posing challenges for the analysis of observed behaviour, self-reports.
*''[[Sensemaking|Information Foraging, Sensemaking, Insight, & Serendipity]]'': (see Dervin (1992)), involves internal constructions / mental models that lead to awareness of knowledge gaps and problem reconstruction, driving information seeking behaviour. Derived from the philosophy and learning theory of John Dewey (1960). A process-oriented theory in which individuals accrue knowledge and also insights, opinions, intuitions, evaluations, and effective responses.
*''Media Use as Social Action'': (MASA, see Renckstorf and ~McQuail (1996)) information acquisition is largely determined/motivated by social behaviour; distinguishes between social, instrumental, and intrinsic use of media.
*''Play Theory'': (see <<cite Stephenson1967>>)) people tend to choose entertainment over information. Even in serious work people prefer information to be presented in a stimulating format; people mix work and play. Helps to explain contexts or tools that encourage creativity. It also explains the use of media for entertainment, diversion, and fodder for conversation. Information and entertainment are inextricably entangled. This may explain motivation for open-ended exploration, where there is no "need" (see Toms (1999)):
>There was no need, no anomalous state of knowledge and no knowledge gap evident. This was simply an information gathering experience without expectations or predicted outcome; novelty stimulated curiosity (and thus exploration).
!!!!Comments & Questions
*Many theories of information behaviour refer to passive media, such as TV. How do they explain interaction? They do account for motivations and expected utility, but not of explaining use. Though it is evident that people avoid making any sort of effort.
*Hard to disentangle ''uses and gratifications'' and ''play theory''
!![Casner/Larkin1989] - cognitive efficiency considerations for graphic design
<<cite Casner1989>>
!!!!Comments & Questions
*
!![Casner1991] - ~Task-analytic approach / BOZ
<<cite Casner1991>> (and <<cite Casner1991a>>) describes a task specification system for automatic graphical presentations, or non-interactive information visualization.
The overall high-level intent of graphical presentations is to facilitate and simplify expensive cognitive work, these being computation and search. With the former, a graphical presentation allows a user to substitute a perceptual inference for a difficult logical inference. With the latter, a graphical presentation can support pre-attentive and often parallel visual search, exploiting spatial locality and Gestalt principles, thereby reducing search time. Perceptual tasks are efficient and accurate. In both cases, the authors draw from <<cite Larkin1987>>'s research in cognitive science (why a picture is worth 10,000 words), as well as <<cite Koedinger1990>> who demonstrated how graphical presentation improve recall for presented data, reduce short-term memory loads during problem solving, provide information about problem state, help users organize their knowledge, and increase motivation and satisfaction.
Computation and search tasks can be defined by logical procedures, which can then be translated into perceptual operators and rendered as graphics. These logical procedures can be defined by domain information (names, types, and values), logical operators (computation and search), and a procedure body: a sequence of operators performed to original domain information and intermediary logical facts, derived information being the results of previous operations.
The process of task specification maps logical operators (to be performed on raw data values) to perceptual operators (to be performed on visual encodings of position, size, shape, shading, color, label, symbol, slope), both enabling the following tasks:
*''Search'' operators
**''search'': given a graphical property, search for objects having that property
**''lookup'': given an object, determine a property of that object
**''search and lookup'': search for an object and determine a property of that object
**''verify'': given and object and property, verify that the object has that property
*''Computation'' operators
**''equal'': do objects have same graphical property?
**''less than'': does one object have a lower valued graphical property than another?
**''greater than'': does one object have a greater valued graphical property than another?
**''plus'': compute the sum of two graphical properties
**''difference'': compute the difference of two graphical properties
**''times'': compute the product of two graphical properties
**''quotient'': compute the quotient of two graphical properties
!!!!Comments & Questions
*uses a running example of using graphical presentations for airline booking
*Authors state that graphic design wisdom (Tufte, Bertin, Cleveland) does not articulate why graphics succeed, the focus is on information to be presented and does not concern itself for the tasks intended to be supported. Is the task not implicit in the design? Does Tufte really not discuss tasks?
*search operators overlap significantly, and nest within each other
*computation operators ''equal, less/greater than'' are comparison operations
*while ''computation'' and ''search'' are high-level tasks, the perceptual operators are low-level, non-interactive tasks
*possible mid-level tasks: compare, derive (aggregate computation operators), filter, sort, (aggregate search operators) - inferred from examples in Table VIII
*<<cite Koedinger1990>> tasks allude to high-level tasks: recall information, problem solving, retrieving information about problem state, organizing knowledge
!![Chang2009] - Defining Insight
<<cite Chang2009>> discerns between two definitions of insight and the implications for these two definitions for the design and evaluation of visualization and VA tools. One arises from the cognitive science literature: that insight is an event, a "eureka" or "a-ha" moment, a change of paradigms and brought on by looking at a problem in a new way. They have neurological evidence for different brain activation during these moments of spontaneous insight. The other definition arises from VA and visualization research that characterizes insight as a quantity, an amount of knowledge gained that occurs upon integrating and building upon one's existing representations, making associations between disparate concepts. While neither is trivial to track or measure, the authors suggest that in the context of visualization and VA, the two forms of insight, however distinct as they are, support each other, occurring in a loop, wherein the knowledge-based insight elicits or enables event-based insight.
!!!!Comments & Questions
*If one form of insight can be measured, the other can be inferred?
*A good starting point into more of the cognitive science literature on insight; insufficient detail here - nothing eye-opening. (citing a New Yorker article?)
!![Chen2009] - Data, information, and knowledge in Visualization
<<cite Chen2009>>
!!!!Comments & Questions
*
!![Chi1998]
<<cite Chi1998>> defines an //operation// as a user interaction; the authors propose an //operator// framework that follows Card and Mackinlay's information visualization pipeline (1997), which delineates the value space from view space. Thus interactions/operations may occur at various stages between the source data (value space) and resulting visualization (view space), with some operations affecting both (brushing and linking, for instance); and introducing subtle distinctions between potentially ambiguous operations such as ''filtering'', which could mean ''view filtering'' or ''value filtering''. Operators at some stages of the pipeline (the view space more so than the value space) are more generalizable and reusable across domains, such as navigation operations. View space operators tend to be more interactive
The authors also distinguish operationally similar operators and functionally similar operators, with the latter not necessarily requiring operational similarity: different filtering operations, for instance, are functionally similar (reducing a view scope or set of values, but not necessarily operationally similar (slider ranges vs. lassoing vs. spreadsheet cell selection).
The operator pipeline described in later sections is one based on the original Card and Mackinlay pipeline, however with the potential for many operators within and between stages, to account for multi-form, multi-window visualization applications.
They classify operators at each stage of the visualization pipeline:
*''data stage'': value filtering, subsetting,
**''data transformation'': deriving/computing new attributes / aggregating
*''analytical abstraction stage'': select subset to visualize
**''visualization transformation'': e.g. cluster, MDS,
*''visualization abstraction stage'': simplify
**''visual mapping transformation'': choose visual encoding technique
*''view stage'': navigation (rotation, translation, scale, zoom), orientation, view-filter, dynamic view-filtering, brushing, animating, focus, permute
!!!!Comments & Questions
*Applies to ~SciVis just as it does to ~InfoVis
*example/technique driven, not very helpful for classifying low-level tasks within the high-level stages
*does not tie into high-level user goals / high-level tasks: what is the user trying to do not addressed
*stages are cyclical, nonlinear
*''operand-based taxonomy'': data, analytical abstraction, visualization abstraction, view (also in <<cite Chi2000>>)
**in <<cite Roth2012>> meta-taxonomy
!![Chuah/Roth1996] - semantics of interactive visualizations
<<cite Chuah1996>> distinguish between lexical, syntactic, and semantic levels of interaction, the latter being the focus of this article. The ''lexical'' level refers to hardware input and output, and the ''syntactic'' level refers to sequences of low-level interaction without regard to their semantic meaning. The ''semantic'' level refers to the intent or meaning of a sequence of low-level interactions. They refer to the basic unit of a semantic interaction as a ''basic visualization interaction'' or ''BVI'', which has inputs and outputs. Inputs are in the form of either basic or composite interaction tasks (user input) or default specifications, and conditions/parameters for the operation: control objects (data objects, graphic objects, virtual device objects/widgets), values (ranges), attributes (graphic, data, state), formulas (multivariable relationships), and focus sets (predefined or user-selected, objects or space). Outputs take the form of changes to the control state of the application or the underlying data state, by adding, deleting, or changing which data is shown, deriving new data. Both of which in turn affect the graphical state of visually represented data - affecting individual or sets of objects or attributes. ~BVIs can be broadly classified as graphical operations, set operations, or data operations. ~BVIs can subsequently be concatenated into higher-level tasks, which can be subject to order-of-operation constraints and resource conflicts.
Basic Visualization Tasks:
*''graphical operations'': change appearance of visualization / graphical representation
**''encode data'': change / transform mapping b/w data and graphical representation, relying upon the values in the underlying data
***''create mapping'': altering graphical ''object'' / ''attribute'' mappings
***''transform mapping'': ''shifting'' the encoding range to separate out sets of objects or ''rescaling'' the encoding range to magnify differences among values for a particular attribute
**''set graphical value'': alter visual representations of a set of entities by uniformly transforming attribute values to some ''constant'' value or according to a formula (''graphical transform''), which can either be continuous or non-continuous. Not related to the underlying data, and thus can be applied to either objects or spaces
**''manipulate objects'': ''creating'', ''copying'', and ''deleting'' graphical objects - doesn't affect encoding of attributes or the underlying data
*''set operations'': create, summarize, join, delete manipulate object sets (classify)
**''create'' set; (group), aggregate objects together
***''enumerate'' objects
***''express membership'' of an object and attribute or an attribute and a value or value range, perhaps via some formula or constraint
**''delete'' set (ungroup or batch delete?)
**''summarize'' set according to some attribute and formula
*''data operations'': manipulate data encoded in visualization, create simulations or what-if analyses; augment the data with new findings
**''add'' objects
**''delete'' objects
**''derive attributes'' for objects according to a set of attributes and a formula
!!!!Comments & Questions
*Paper centres around a multiform, multivariable real-estate visualization example
*"during analysis, users usually discover new facts about the data" - how is this accomplished using the BVI primitives?
*still fairly low-level, even at it's highest categorization (manipulate graphical objects, manipulate sets, manipulate data) is still far removed from user goals / intents
*does ''set-operation / delete'' mean delete entities forming the set or to ''disassociate'' the entities from one another, dissolving the set?
*''transform mapping'' and ''graphical transform'' are hard to distinguish
*not a very comprehensive related/previous work section, which deals predominantly with lexical and syntactic taxonomies
*in <<cite Roth2012>> meta-taxonomy
!![Cleveland1984] - raphical perception: Theory, experimentation, and application to the development of graphical method
<<cite Cleveland1984>>
!!!!Comments & Questions
*
!![Cottam2012] - ~VisWeek '12 Dynamic Vis Taxonomy
<<cite Cottam2012a>>: Joseph Cottam, Andrew Lumsdaine, Chris Weaver:
*dynamic changes occurring throughout the visualization pipeline (e.g. streaming data)
*"if you're working at the level of items changing on the screen, it can sound a little funny" - [[Onion video about concentric circles|http://www.theonion.com/video/breaking-news-series-of-concentric-circles-emanati,14204/]]
*a taxonomy, a vocabulary for what is occurring on the screen with dynamic data
*quantifying/spatial categories (fixed, mutate, create, create/delete) and retinal categories (immutable, known scale, extreme bins, mutable scale) (Bertin): categories of dynamic change: identity preserving, transitional, immediate
**identify preserving transformations: fixed x [all], mutate x immutable
**transitional: mutate x [known scale, extreme bins], create x [immutable, known scale, extreme bins]
**immediate: mutate x mutable scale, create x mutable scale, create/delete x [all]
*type of transformation depends on task, decision tree for task
*examples: Gapminder, flow visualization, taxicab visualizations from NYC, London
*varying time scales, works best for streaming and dynamic queries, but not so well for reloading a new dataset, or interactivity
!!!!Comments & Questions
*decision tree of tasks: cite/read for task work
*how was the taxonomy validated?
!!!!Questions
*''Enrico Bertini'' (NYU Poly): (1) methodology followed to come up with model? (2) how do imagine people use your model?
*(1) we had a lot of videos, and categorized them, inter-coder reliability, agreement of categories, resolved ambiguities (2) it introduces a well-defined vocabulary, decision tree for tasks/process/questions for designers: guidelines
*''George Grinstein'': re: assumption that retinal and spatial variables are not independent. Taxonomy is not that clear cut. This is good first cut, but the two dimensions are so interrelated.
**We do recognize the ambiguities and redundancies that exist in the paper. "I'm going to hide behind Bertin who said that the two dimensions were separable enough to treat them that way."
!![Craft2005] - Beyond the Visual Information Seeking Mantra
<<cite Craft2005>>
!!!!Comments & Questions
*
!![Crampton2002] - Interactivity types in ~GeoVis
<<cite Crampton2002>>: ''objective-based taxonomy'': examine, compare, (re)order/(re)sort, extract/suppress, cause/effect; ''operand-based taxonomy'': data, representation, temporal dimension, contextualizing interaction
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
!![Crouser2012] - ~VisWeek '12 Affordance framework for VA
<<cite Crouser2012>> present an affordance-based framework for mixed-initiative systems (human-computer collaboration systems), distinguishing human affordances and machine affordances, choosing not to take a task-centric or deficiency-centric approach to constructing the framework. In other words, rather than allocate tasks to a human because a computer can't do it (computationally intractable), it takes the view that tasks should be allocated to the human because the human is good at it. The recognize that humans adapt and can learn complex tasks. Human and computer should leverage the affordances of the other for harmonious cooperation and problem-solving.
Human affordances and potential tasks:
*''visual perception'': scan, recognize, and recall images; detect changes in size, shape, colour, movement; accurate pointing and dragging objects; augment image labels / annotations; visual search;
*''visuospatial thinking'': reason about (complex) spatial relationships;
*''audiolinguistic ability'': augment computer vision and OCR; tagging; audio transcription;
*''sociocultural awareness'':
*''creativity'': augment automated systems; outlier detection;
*''domain expertise'': diagnosis; classification; domain-specific transformations; infer trends;
Machine affordances and potential tasks:
*''large-scale data manipulation'': refine classification; DR; interactive clustering; extract transfer functions; suggest informative views; externalize insight generation process (provenance and workflow)
*''collecting and storing large amounts of data'': aggregate and store
*''efficient data movement'' (in time and space): augment visual processing / exploration; facilitate access to related information/data;
*''bias-free analysis'': prediction; propose candidate visualizations; dissimilarity highlighting
They use examples to define the framework, and include two case examples of systems that leverage multiple human and machine affordances.
The framework can be used for guiding function (task) allocation: to human, computer, or both. Using the framework predictably is still difficult.
In evaluating human computational tasks, we still need to understand the relative and absolute costs and complexity of underlying mechanisms. We can leverage existing evaluation protocols from computer science: //"much of the standard theoretical language holds true, including algorithmic complexity, problem complexity, complexity classes and more"//.
We need to //"to develop a set of canonical actions that humans can perform with known complexity, but compiling this list is nontrivial"//. Individual differences must also be understood. Their conclusion bodes well for the future of theoretical work in VA/~InfoVis:
>//While there has been remarkable progress in the development of novel solutions to support analytic processes, we have not yet fully realized our potential as a systematic science that builds and organizes knowledge in the form of testable theories and predictions.//
!!!!Notes:
*[[slides|http://goo.gl/ujLGC]]
*meta-review of human-computer-collaboration (HCC)
*curated 49 papers / 1,300 from various HCI/Vis venues
*framework on affordances: affordances b/w agents and machines: computers rely on humans to provide visual perception, visuospatial thinking (e.g. Fold It), audio-linguistic ability, sociocultural awareness, creativity, domain knowledge; humans rely on machines for large scale data manipulation, collection and storage of data, efficient data movement, bias free analysis; HCC systems involve multiple affordance combinations; many systems leverage the same affordance sets: a possibility to compare alternative systems
*examples: Fold It, ~PatVis (Koch '09)
*guidance to leverage affordances, a framework for evaluation, prevent overload: we need to quantify affordances and workload
*Shahaf / Amir '07: human-assisted turing machine
!!!!Comments & Questions
*relationship b/w affordances and tasks; (1:1) (N:1) (N:N)?
*contribution: common language (a list?)
*cite for task paper
*a descriptive, not prescriptive framework, theoretical comparisons of affordance and tasks/functions still can't be accomplished.
!!!!Questions
*''Q'' why not citing work related to automated feedback algorithm for annotating image?
**Not aware of this work.
*''Tim Lebo'' (RPI): what is the common language? Is it a task list?
**language: affordance slides; not tasks, specific affordances, a set of skills that can be applied to multiple tasks
*''Eli Brown'' (Tufts): where are interesting spaces in affordance list?
**some interactions of affordances are not explored e.g. sociocultural knowledge and large scale computation
!![Dadzie2009] - Visual knowledge exploration
<<cite Dadzie2009>>
!!!!Comments & Questions
*
!![Dibiase1990] - EDA for Vis
<<cite Dibiase1990>> extends <<cite Tukey1977>>, <<cite Tukey1980>>, distinguishing ''visual thinking'' from ''visual communication''. The former occurs at ''early stages'' of scientific inquiry, involving many processes ("solutions") of ''exploration'' (examining data from multiple perspectives, hypothesis generation, forming research questions) and ''confirmation'' (hypothesis validation, formal hypothesis testing) (<<cite Tukey1980>>). The latter involves stages of ''synthesis'' (integrating ''insights'' and ''triangulating'' on a final solution to research questions ) - Dibiase's addition to EDA, and ''presentation'' (communicating). A diagram in <<cite Roth2012>> illustrates this (the Swoopy diagram, fig. 5; reflecting the number of solutions over time/stage)
!!!!Comments & Questions
*Cited / discussed in <<cite Roth2012>>: high-level goals are exploration, confirmation, synthesis, presentation.
*Original article not online, not available via UBC library
!![Dix/Ellis1998] - Adding simple interaction
<<cite Dix1998>>: ''operator-based taxonomy'': highlight and focus, accessing extra information, overview and context, same representation-changing parameters, same data- changing representation, linking representations
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
*in <<cite Yi2007>> RW
!![D沫11] - Information Flaneur
<<cite Dork2011>>'s notion of the //information flaneur// likens to the curious urban explorer, at least in an experiential way, who is provoked to pursue multiple diverging paths through an intformation space.
!!!!Comments & Questions
*flaneur (def): lounger, idler
*TM comment May 29 / '13
>//Also to consider is the information flaneur paper from the group that they self-cite, as part of our discussion of the play stuff.//
*We could cite D沫11 (Info Flaneur) in the body text when defining 'enjoy' and 'explore', though I think that our Sprague et al. ref already provides enough support for the idea that there exists users who "enjoy / explore / stroll".
!![D沫2011a] - explicit and implicit relations of complex info. spaces
<<cite Dork2011a>>
!!!!Comments & Questions
*
!![D沫2012] - ~PivotPaths / faceted information spaces
<<cite Dork2012>> distinguishes between ''searching'', ''browsing'', and ''visualization'', as well as their definition of ''strolling'' in Dz:
*''searching'': finding the most relevant resources, iterative filtering from many to few, a narrowing or funneling down, usually resulting in a single find
*''browsing'': navigation from one resource to the next via hyperlink of connection, more open-ended than searching, still at the level of individual resources; it is difficult to put resources into context and determine what attributes they share.
*''visualization'': analyzing a collection of resources at a high level, differing from searching and browsing, starting at a higher overview level, allowing the user to see global patterns; resource-level investigation is difficult, typically it is not casual enough, exploratory (cites the information flaneur paper, <<cite Dork2011>>)
*''strolling'':
>"//the casual exploration of an information space using interactive visualizations that offer a wider perspective on relations and facets, and provide diverse pathways for gradually pivoting between sets of related resources.//"
!!!!Comments & Questions
*TM comment May 29 / '13
>//I just ended up reading this paper more closely - do you think the discussion of search vs browsing in Section 3.1 merits citing it in the task taxonomy paper?//
*It's worth citing D沫2012 in the table, but not in the text, because I don't find their definitions for browse or visualization to be particularly useful. Their definitions map to ours as follows:
>them -> us
>
>search -> locate (target known, location unknown)
>browse* -> browse (target unknown, location known)
>visualization** -> summarize
>stroll -> explore (target unknown, location unknown)
>
** their definition of browse is tightly coupled with the experience using a web browser, and not to broader definitions of the word, such as in Marchionini '06. For instance, what about browsing in a bookstore?
** I find their definition of "visualization as overview" to be odd, ignoring Shneiderman's mantra and the use of interactive visualization for seeing both overview and detail.
!![Dou2009] - Recovering Reasoning Processes from User Interactions
<<cite Dou2009>> conducted a study in which a team of 4 volunteers (recruited from the university, familiar with a financial analysis tool) coded interaction logs of visual analysts of financial data. The visual analysts' analysis sessions were videotaped and they were encouraged to use the [[Think Aloud Protocol]] as they worked; this established the ground truth for what user's did: the video and utterances were coded into 3 distinct categories:
*''finding'': a decision made after a discovery
*''strategy'': the means to arrive at a ''finding''
*''method'': the link between ''finding'' and ''strategy'', the steps for implementing a strategy
The results indicate that the 4 independent volunteers, familiar with the tool, were able to accurately and reliably code analysts' interaction logs into these categories: finding, strategy, method, matching the ''ground truth'' of the video/think-aloud recording.
!!!!Comments & Questions
*''strategy'' ~ ''methodology''?
*One would hope that a finding (a noun) could be categorized distinctly from a method or strategy. The subtle distinction is at what point does a method become a strategy - as described, they are hierarchical, and a method is an implemented strategy, thereby less abstract. And yet the relationship between strategy and method also recalls the relationship between methodology and method, in which methodology is not an abstraction of a method, but a recipe or ordering of methods. How did coders interpret the meaning of strategy? Not surprisingly, coding accuracy for ''strategy'' and ''method'' were less than accuracy for ''finding''.
*How did they overcome subjective self-reporting biases of the analysts, analysts' retrospective biases, and the biases of video/think-aloud coders?
*What is the granularity of finding, strategy, and method (the steps of the implemented strategy)?
!![Duke2005] - "Do you see what I mean?"
<<cite Duke2005>>
!!!!Comments & Questions
*
!![Eccles2008] - Stories in ~GeoTime
<<cite Eccles2008>>
!!!!Comments & Questions
*
!![Edsall1995] - interactivity in Spatial data
<<cite Edsall1995>>: ''operator-based taxonomy'': zooming, panning/re- centering, re-projecting, accessing exact data, focusing, altering representation type, altering symbolization, posing queries, toggling visibility, brushing and linking, conditioning
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
!![Ellis1997] - information seeking patterns of engineers/scientists - LIBR
<<cite Ellis1997>>
!!!!Comments & Questions
*
!![Espinosa1999] - Domain analysis
<<cite Espinosa1999>>
!!!!Comments & Questions
*
!![Fekete2011] - Obvious - Vis toolkit to bind them all
<<cite Fekete2011>>
!!!!Comments & Questions
*
!![Fisher1993] - Map design and visualization
<<cite Fisher1993>>
!!!!Comments & Questions
*
!![Foster2003] - serendipity and information seeking - LIBR
<<cite Foster2003>>
!!!!Comments & Questions
*
!![Friel2001] - Making sense of graphs
<<cite Friel2001>> describes 2 high-level uses of graphs: ''translation'' (''locating'') and ''interpretation'' / ''integrating'' (involving ''re-arranging'', ''sorting'', [''filtering''], and its ''generative'' extensions ''interpolation'' and ''extrapolation'').
Cites Graesser et al (1996) and the analog of the comprehension of text, discerning between low-level questions and deep questions of ''inference'', ''application'', ''synthesis'', and ''evaluation'' - identifying gaps, contradicitons, incogruities, anomalies, ambiguities.
In the vocabulary that <<cite Mayr2010>> and <<cite Pohl2012a>> picked up on, there are elementary questions (''extract information from the data'') / focusing on one quantity, intermediate questions (''finding relationships in the data''), and overall questions (''moving beyond the data'') / focusing on integrating information acoss data points / making comparisons / identifying trends /.
On the characteristics of tasks, Friel et al refer largely to the low-level graphical perception judgment tasks of Cleveland/McGill: comparison and proportion judgments.
!!!!Comments & Questions
*Like Chuah/Roth, they refer to the ''semantic'' (context)/''syntactic'' distinction in (static) graph interaction.
*Corroborates with graphical perception work of Lohse (1993) and Graham (1987): describing data vs. summarzing, comparing, generalizing, predicting
!![Fry] - seven stage process of visualizing data
In //Visualizing Data: Exploring and Explaining data with the Processing environment//, <<cite Fry2008>> describes a seven stage process of visualizing data:
*''acquire'': obtain data
*''parse'': provide structure to data (e.g. order)
*''filter'': remove all but data of interest
*''mine'': apply stat. methods / data mining to discern patterns
*''represent'': choose basic visual model
*''refine'': improve reprentation, make clearer, make more engaging
*''interact'': add methods for manipulating data or controlling what features are visible
!!!!Comments & Questions
*
!![Gawron1989] - A taxonomy of independent human performance
<<cite Gawron1989>>
!!!!Comments & Questions
*
!![Gleicher2012] - evaluation arms race - actionable and persuasive (BELIV '12)
<<cite Gleicher2012>>'s //Stop The Evaluation Arms Race! A Call to Evaluate Visualization Evaluation//
!!!!BELIV '12 Notes:
*Why evaluate? reflect a change in thinking. Why ask why? Not whether evaluation is necessary, but what we should evaluate, how evaluation can be done well. Good evaluations can guide and persuade. Are guiding and persuading diamteric opposites?
*guide: non-prescriptive -> actionable
*persuade: vacuous assertion -> compelling argument
*how do you (inform/convince) the (audience) to do some (thing); audience, thing = context
*evaluating evaluations: making them good: actionable, persuasive
!!![Glueck2013] - model for navigating large data views
<<cite Glueck2013>> proc. GI '13
!!!!Comments & Questions
*
!![Gotz/Zhou2006] - characterizing information gathering / results processing in intelligence analysis
<<cite Gotz2006b>>
!!!!Comments & Questions
*
!![Gotz/Zhou2008] - Characterizing users analytic activity
<<cite Gotz2008>> characterize visualization at 4 levels of granularity. The highest level is that of domain-specific, highly semantic ''tasks''; then there is a level of ''subtasks'', often domain-specific, yet more concrete and objective, however these can be described in a domain-independent fashion, such as <<cite Amar2004>>'s //Analytic Gaps//. Subtasks break down into semantic yet domain-independent and generic ''actions'', which are categorized as exploratory actions, insight actions, or meta-actions, each carrying a type of user intent. Up to this point, tasks, subtasks, and actions are visualization and interaction independent. At the lowest level are ''events'', which are syntactic and visualization-specific interactions, wherein the semantics (user intents) of the events cannot be recovered. Their taxonomy was motivated and inspired by [[Activity Theory]], "which describes how tools act as mediators of human activity".
The authors focus on the ''action'' level, which are both domain and visualization tool/interface independent, the building blocks to insight provenance. Actions are mid-level and bridge the gap between high-level cognitive activity and low-level events. Actions are characterized by type, intent, and a set of parameters.
They recruited 30 users for an observational study. Users were given an hour to perform one of two domain-specific ''tasks'' with a commercial visual analytics tool (analysis of stock market data for investment opportunities with [[Map of the Market tool from SmartMoney|http://smartmoney.com]] or business travel planning task with a map-based travel visualization.
The authors observed a hierarchical decomposition of tasks from higher-level cognitive activities to lower-level concrete user actions. User ''actions'' were task- and tool-independent; the intent behind these actions were the same regardless of when and where they occurred. Among actions, there tended to be a common progression, which the authors categorized into phases: exploration and insight. Actions are independent to underlying visual metaphors and interaction events that support or result from the action.
Their action taxonomy (based on their observations and the afforded interactions of several visual analytics systems) is as follows (Adapted from Fig.2, Table 1):
*//exploration// actions: searching for insights; intent is data-change and/or visual change
**''data exploration'': //data change// intent
***''filter'': //data/visual change//, reduce the data set being visualized
***''inspect'': //data/visual change//, ''details-on-demand'' request for a visual objects
***''query'': //data/visual change//, request to bring additional data into visualization
***''restore'': //data/visual change//, restore a previously save bookmark
**''visual exploration'': //visual change// intent
***''brush'': //visual change//, highlight a set of visual objects
***''change-metaphor'': //visual change//, alter active visualization technique (encoding)
***''change-range'': (zoom / pan), //visual change//, scale/scroll a visualization to display a new range along an ordinal dimension
***''merge'': //visual change//, combine visual objects into a single visual group
***''sort'': //visual change//, re-arrange the visual presentation along a particular dimension
***''split'': //visual change//, break apart the data represented by visual objects into new several visual objects
*//insight// actions: //notes change// intent, manipulating discovered insights
**''visual insight'': parameters refer to visual objects
***''annotate'': //notes change//, tag meta-information to the data represented by visual objects
***''bookmark'': //notes change//, save current visualization for future review
**''knowledge insight'': manipulation of new knowledge as a result of knowledge synthesis
***''create'': //notes change//, create a new note entry
***''modify'': //notes change//, alter an existing note entry
***''remove'': //notes change//, delete an existing note entry
*//meta// actions: //history change// intent, distills units of activity (domain- and tool-independent actions)
**''delete'': //history change//, removes an action from a sequence of performed actions
**''edit'': //history change//, modify parameters of a previously performed action
**''redo'': //history change//, re-perform an undone action
**''revisit'': //history change//, return to earlier stage of analysis
**''undo'': //history change//, remove most recently performed action
They later validated/used this taxonomy/characterization to develop a visual analytics system called HARVEST, which explicitly used action names, types, and parameters in the interface.
!!!!Comments & Questions
*"relatively little attention has been focused on characterizing actions" - I think there's still a lot of overlap between their action level and the task taxonomies of previous work listed below, at least in terms of terminology and tool-independent-ness.
*authors claim that previous task taxonomies (<<cite Amar2005>>, <<cite Chuah1996>>, <<cite Chi1998>>, <<cite Wehrend1990>>, <<cite Zhou1998>>) were system-oriented ,focusing on system response to user activity and not on user activity itself; true? would these be considered low-level, despite being interface/visualization tool independent?
*authors also claim that other previous taxonomies (<<cite Amar2004>>, <<cite Gotz2006b>>, <<cite Shneiderman1996>>) are too high-level, equivalent to their level of tasks and subtasks; however, these task taxonomies are domain-independent - subtasks can be expressed in a domain-independent way.
*Characterization more complicated than it needs to be? Many levels and cross-cutting aspects and terminology to this characterization: tasks, subtasks, actions, events, intents, parameters, types, categories / stages / phases, trails phases of action (exploration, insight) maps to sub-task (these are called trails?); trails can be represented by regular expressions of exploration and insight actions
*Does not define what ''insight'' is, or how one moves from exploration phase to insight phase; what is ''knowledge synthesis''? in the ''knowledge insight'' action characterization?
*knowledge-insight and meta actions are not specific to visualization (is this a VA / ~InfoVis disagreement?)
*Deciding upon the observational study tasks a priori dictates which subtasks and actions take place
*Map-based travel tool in observational study not specified
!![Green/Fisher2009] - personal equation of complex individual cognition
<<cite Green2009a>>
!!!!Comments & Questions
*
!![Hampson2005] - Instrumenting the intelligence analysis process
<<cite Hampson2005>>
!!!!Comments & Questions
*
!![Heer/Shneiderman2012] - Interactive Dynamics
<<cite Heer2012a>> present a list (taxonomy) of user tasks (interactive dynamics) that define what users do with visualization, from initial data acquisition, wrangling, cleaning, and transforming - to sharing, presentation, and collaborative decision making. They structure their tasks as follows, with 3 categories and 12 tasks (4 in each category), however many finer-grained tasks and operations occur within these tasks. Examples of systems or techniques are given.
*//data/view specification//
**''visualize'': indicate which data to be shown / how to show it (e.g. //Tableau/Polaris//)
**''filter'': shift focus b/w data subsets (e.g. //Spotfire//, //Google Hotel Search//)
**''sort'': ordering to surface trends, clusters, organize by familiar unit of analysis: time, financial quarter, etc. sophisticated sorting by //seriation// (e.g. reorderable matrices)
**''derive'': transform variables, create attributes from old ones, summarize and aggregate input data, gather descriptive stats; automation of the derivation process and view creation using pattern recognition, applying statistical models
*//view manipulation//: highlight patterns, drill down, select
**''select'': highlighting, manipulating individual objects - annotation, details-on-demand, closely linked to filtering; selection as query (e.g. selecting individual time-series lines in //~TimeSearcher//, parallel coordinates)
**''navigate'': visual information seeking mantra: overview first, zoom and filter, details on demand; general / global assessment then local / contextual fine-grained analysis; semantic zooming; "search, show context, expand on demand"; viewports (e.g. Focus+Context, Overview+Detail displays, //~DateLens//, DOI trees)
**''coordinate'': see related details or items highlighted in other displays; linking and brushing; marking sets/subgroups and exporting selections; rapid comparisons of different data dimensions or time slices; hiding or showing/highlighting the same items in linked multiform displays, assessing/integrating how patterns project across views (e.g. small multiples, //Improvise//, //~GGobi//)
**''organize'': open, close, maximize, lay out different components of the GUI(s); automatic view layout, automatic resizing; adding/removing views
*//process + provenance//: iterative data exploration and interpretation
**''record'': hypothesis tracking, progress tracking, review, summarization, communication of findings, undo/redo, interaction histories, interaction logging, visual histories of prior states with metadata (e.g. behaviour graph, visual histories in //sense.us//)
**''annotate'': recording, organizing, communicating insights - graphical and text annotations, expressive pointing; data-aware annotations acting as selections, linking and brushing via data-aware annotations
**''share'': social interaction and discussion, view exporting, application bookmarking, publish functions, asynchronous vs. synchronous collaborative analysis, co-located vs. remote collaborative analysis (e.g. //sense.us//)
**''guide'': formalize analysis strategies and present to novice analysts; narrative visualizations: hand-holding step-by-step through a linear narrative, tacit tutorial, data-driven stories; systematic-yet-flexile exploration and progress recording; reusable workflows (e.g. //~SocialAction//, NYT graphics)
!!!!Comments & Questions
*Magazine format, addressed to the layperson reader
*flat/shallow hierarchy of tasks, not a true taxonomy - many have interdependencies, nesting
*insufficient for evaluative power (<<cite Beaudouin-Lafon2004>>) - mixture of high-level abstract cognitive tasks, low-level interface tasks, interaction and data encoding dependencies
*how much does analytic provenance (previsualization tasks, data wrangling and cleaning) need to be part of the mid-level task characterization; how much do sharing and presentation tasks need to be part of the task taxonomy?
*how to incorporate semi-automated statistical methods within a visualization environment?
!![Hetzler2004] - Analysis experiences using information visualization
<<cite Hetzler2004>>
!!!!Comments & Questions
*
!![Heuer1999] - psychology of intelligence analysis - RR
<<cite Heuer1999>>
!!!!Comments & Questions
*
!![Hoffman2004] - The pleasure principle
<<cite Hoffman2004>>
!!!!Comments & Questions
*
!![Hollan2000] - distributed cognition in HCI
<<cite Hollan2000>>'s article is a good introduction to Distributed Cognition (DC) and an associated research framework for HCI. The DC framework breaks from the traditional cognitive science viewpoint that cognition is occurring solely in the mind of the individual; it also breaks from the traditional HCI view that there is a gulf between a system model and mental model that must be crossed by some symbolic transformation or gesture, an interaction that translates a desired mental state into a system state. Instead, DC posits that cognition is embodied both in the minds of individuals, in the functional relationships with elements and artefacts in the environment, and within those artefacts themselves. Instead of a translational gulf, DC states that there is a coordinated set of processes, or trajectories of information, occurring between internal cognitive states and external cognitive states, the latter subject to interactions and communication with others and artefacts in the environment over time. In other words, work materials are more than just stimuli, they are part of the DC system.
Methodologically, studying a DC system requires triangulation between multiple methods: an event-centred cognitive ethnography leveraging social science research methods (interviews, surveys, participant observation, video/audio transcripts) and interaction logs. Establishing rapport is key. Technical expertise in the domain of study, and thus participant observation, is particularly useful and encouraged. This informs the design of targeted research experiments and subsequently new workplace environments and artefacts / tools. Premature design of environments and tools risks destroying "many of the most valuable aspects of current ways of doing things because we do not understand how they work", as exemplified by analog controls in aircraft cockpits, which are used for more than value retrieval; they provide derived values and afford peripheral monitoring to a greater extend than do digital displays.
Their suggested research framework involves studying how people establish and coordinate structure within an environment, how coordination is maintained over time, how cognitive effort is offloaded to the environment when it is practical to do so, achieving a better conceptualization of what is going on and what ought to be done, and when and how cognitive load-balancing is achieved between human actors, the environment and its artefacts. In short, we need graduates with a wider range of methodological skills.
Several examples follow, including: the study of aircraft cockpits, the bridges of ships, direct manipulation interfaces (emphasizing that some screen-space objects and events (e.g. icons) have no physical correlate (physical files), as in moving and arranging icons on a desktop, and yet the "location" of these files forms part of our cognitive system), history-enhanced digital objects, and Pad++ (semantic zoom interface).
Implications for design include ensuring that representations facilitate flexible use, making representations serve as indicators as to what to do next, providing a history of use so as to inform future interactions. Realizing that space is a precious resource, like time and memory. Environments should be modified with the goal of cueing faster recall, generating mental images faster, provide scaffolding.
!!!!Comments & Questions
*cited by <<cite Liu2008>>, in particular, w.r.t. DC in ~InfoVis
*"design destroying current ways of doing things" could be reflective of unacknowledged affordances and implicit interaction, interaction that cannot be recovered from a [[Think Aloud Protocol]]
*a paper to cite as justification for longitudinal pre-design and post-deployment field studies
*Doesn't discuss / discern epistemic and pragmatic actions (<<cite Kirsh1994>>), doesn't speak about individually discernible tasks; all activity seems wrapped up in the ongoing process of coordinating internal and external states
!![Horvitz1999] - mixed-initiative systems
<<cite Horvitz1999>>
!!!!Comments & Questions
*
!![Huron2014a] - constructing visual representations w/ tangible tokens (~InfoVis '14)
<<cite Huron2014a>> recruited 12 participants to construct visualizations using coloured wooden tile tokens, based on fictional personal finance data and a prompt relating to helping a friend sort out their personal finances and budget. Researchers coded observed behaviour and later asked participants to update their constructed visualization with new data. Participants were also interviewed about their constructed visualizations. The researchers generated a model to explain the visualization encoding process of mapping data to a constructed representation, as follows:
*1. ''load data'': //read data// (logical task), //compute token(s)/construct(s)// (logical task), //select colors// (logical task), //grasp tokens//
*2. ''build constructs'': //organize constructs//, //create constructs//
*3. ''combine constructs'': //arrange constructs//, //align constructs//, //rotate constructs//, //split constructs//, //move constructs//, //merge constructs//, //optimize constructs//
*4. ''correct'': //remove constructs//, //increase constructs//, //decrease constructs//
*(all other tasks are physical tasks)
!!!!Comments & Questions
*claimed contribution is one of ''unpacking the visual mapping process''
!![Hutchins1995] - cognition in the wild (distributed cognition)
<<cite Hutchins1995>>
!!!!Comments & Questions
*
!![Isenberg2008] - exploratory study of collaborative visual information analysis
<<cite Isenberg2008a>> conducted an observational laboratory study of individual and collaborative visual information analysis. This study resulted in design implications / heuristics for designing and evaluating Information Visualization tools. General processed were studied rather than low-level perceptual or interactive tasks. The observation study did not involve an existing visualization tool, but shared printed materials (graphs and charts of toy data sets). They sought to understand these processes rather than interactions with existing tools.
''Methodology'': a brief tutorial of the research materials and tasks were provided. Individuals were instructed to use the [[Talk Aloud Protocol]]. Group discussion was recorded and analyzed similarly. They were instructed to approach the tasks however they saw fit.
''Findings'': the researchers' observations revealed that information analysis is not temporally organized in consistent ways within or between groups, but that individual sub-processes were discernible: browse, parse problem (if one is defined), discuss collaboration style (complete task division, independent and parallel, joint work), establish task strategy, clarify extracted information, select / extract, operate (most time-consuming - higher-level cognitive work on the data), validate / confirm solution. Groups were slower but more accurate.
Comparisons are made with the [[Information Foraging and Sensemaking]] cycle and models of collaborative analysis. They found their findings to fit well with these models, however discussion of collaboration style was not explicitly represented in these other models.
''Implications'': The analyses processes were temporally flexible, as was collaboration style. Designs should support this criteria, and personal/group workspace organization. Do not assume a standardized temporal flow of one sub-process to another.
Observed interactions:
*Direct interactions with a visualization: ''browse, parse, select, operate''.
*Interactions around an artifact (without necessarily physical contact): ''discuss collaboration style, establish task strategy, clarify, validate''
These later interaction are reminiscent of those found in <<cite Card1999>>'s sensemaking loop.
!!!!Comments & Questions
*Focused and open-ended analysis problems were not specific to a domain. Non-domain users with no specialized knowledge ( i.e. undergraduate students). Data sets were SPSS demo materials. Collaborative materials were printed from SPSS. Difficult to generalize results (surprising as this is a CHI paper). 4 groups of singles, pairs, and triples (N = 24).
!![Isenberg2009] - collaborative brushing and linking for document collections
<<cite Isenberg2009>>
!!!!Comments & Questions
*
!![Isenberg2012] - Co-located collaborative visual analytics around a tabletop display
<<cite Isenberg2012>>
!!!!Comments & Questions
*
!![~Jankun-Kelly2002] - model for visualization exploration
<<cite Jankun-Kelly2002>>
!!!!Comments & Questions
*
!![~Jankun-Kelly2007] - Framework for Vis. Exploration
<<cite Jankun-Kelly2007>> proposes a formal model to explain visualization exploration, a grammar (derivation calculus) for describing the model, and contributes a software framework for recording exploration. Recorded explorations can be shared, analyzed, and encapsulated within subsequent explorations.
In short, visualization exploration is described as a user ''iterating'' through an analysis cycle of visualization ''parameter adjustment'', leading to visualization ''transformation'', producing new ''results'' (see fig. 1). These repeated three steps comprise and exploration session, which can be described by the model and captured by the software framework. These parameters (adopted from <<cite Upson1989>> and <<cite Card1999>> models) include those that control ''data filtering, data transformation, visual (primitive) mapping, rendering'', and ''view transformations''. Multiple parameter adjustments means that exploration is akin to navigating a multidimensional parameter space; the cycle of parameter adjustment and results leads to the ideas of ''visualization space paths'' and a ''derivation model'' for parameter settings. Both give way to <<cite Jankun-Kelly2007>>'s model and framework.
>//Our process model lies between the low-level syntactic models and high-level semantic models of user interaction. Specifically, it addresses the ways a user manipulates parameter values to produce results.//
By this, they refer to interactive parameter control, direct manipulation, and function derivations for cyclic exploration.
!!!!Comments & Questions
*Applies to ~SciVis and ~InfoVis
*Existing task taxonomies (<<cite Chi1998>>, <<cite Chuah1996>>, <<cite Shneiderman1996>>, and <<cite Wehrend1990>>) "focus on the goal of the user, not how the visualization was used to achieve those goals"
*I consider their model to an odd mix of low-level and high-level, and too tightly tied to low-level cyclic parameter adjustment; "generate results / derive insight" is too high-level, as this does not refer to the interpretation of results, braking down units of insight, hypothesis formulation/validation, and decision-making; what about comparison? pattern/trend recognition? cluster recognition?
*However, the authors that that <<cite Upson1989>> and <<cite Card1999>>'s models are not fine-grained enough to describe visualization exploration; <<cite Lam2008a>> has the same criticism of <<cite Jankun-Kelly2007>>.
*I will not focus on the formal model grammar or software framework here, which comprises a bulk of the paper
*A strong claim of generalizability
!![Jarvenpaa/Dickson1988] - graphics and managerial decision making
<<cite Jarvenpaa1988>>
!!!!Comments & Questions
*
!![Jennings2011] - Measuring Creativity
<<cite Jennings2011>> presents a quantitative research methodology for measuring exploratory creativity in an aesthetic visual search task. This task represents a broad set of artistic and creative visual tasks, including composition of a still-life photograph or painting.
The authors describe creativity as a search, and discern between //path search// (outcome is known, path uncertain) and //place search// (outcome is uncertain, means for achieving it are relatively straightforward). They argue that artists' outcomes depends on early decisions, seldom a matter of selecting between a few choices. Their intent is to study and make inferences about artists' search strategies as they explore a creative space (or search landscape) over time. This paper documents their experimental paradigm for capturing one's search trajectory (observable choices), allowing for a better understanding of one's search strategy (hidden to the observer, difficult to verbalize).
An artistic composition task involving the placement of a virtual camera and light source around a virtual scene is their task. They argue that this task captures more than insight or divergent thinking, however it is not as ecologically valid as an actual artistic composition task (painting / photographic a scene). However, their task allows for gains in precision, consistency, and accuracy.
Artistic domains tend to be //blind// in that transitions in a foggy search landscape are non systematic, often resulting in trial and error (see Campbell (1960): //Blind Variation and Selective Retention Theory//). Not all blind processes are random (however a purely random process is blind), nor are they brute force. There appears to exist a //metaheuristic// algorithm, a balance between //intensification// and //diversification// of search trajectories. Retention can also be blind: changes that make the solution worse are sometimes retained so as to move away from local maxima in a search space, despite not guaranteed to lead to higher maxima.
Each participant's search trajectory can be monitored, and their search landscape (ruggedness, fogginess) estimated based on their ratings for target images and their predicted ratings for adjacent (unseen) images. Together these can be used to draw inferences regarding one's search strategy. Their theoretical model is described as: Given a problem (an interface, an open-ended goal, and a scene), an observed search trajectory and solution, and by probing one's goal criteria, we can partially measure the landscape topology. With this information, search strategy (interpretation strategy and exploration strategy) become clearer. This must also take into account ''epistasis'', interactions among goal criteria, such as the conflict between control and evaluation dimensions. Goal criteria can be measured in an open-ended questionnaire and standardized Likert-style survey. Search landscape can be measured by evaluating a standard subset of images (i.e. a 5-by-5 neighborhood) repeatedly in different random orders, which serves to identify local maxima/minima. Creative solutions to the open-ended goals can be evaluated based on novelty (statistical unusualness) and appropriateness. Trajectories can be measured with aggregate metrics for capturing diversification/intensification: proportion of space explored, rate of exploration, instances of doubling-back, and changes in rate. The researchers are also developing a parsing language for quantifying the criterion space of creators.
They have preliminary results, mapping goals to criteria, which in turn affects the search landscape; taken together, these can be connected to diversification and intensification in one's search trajectory, as well as the evaluated outcome. They have begun to analyze goal ratings for open-ended goals on simple and complex scenes, as well as the correlations within and between users' ratings. They have begun efforts to model exploration and interpretation strategies, and are beginning to consider multiple criteria and more open-ended criteria, and how these criteria stabilize or change over time. They are considering studies to induce criteria change explicitly.
The paper concludes with relation to other work (i.e. the "//be creative//" study, in which originality of results was increased when participants were told to be creative), and a summary of limitations of their work. They acknowledge that their work addresses exploratory creativity and not transformational creativity (users do not generate anything new). They are addressing more than insight but not much more than divergent thinking, only one index (albeit a popular index) of creativity. And thus external validity is reduced, however at the gain of a depth of real-time information collected about search behaviour.
!!!!Comments & Questions
*An interesting methodology, curious to see where they take it / expand off of it, and the results they achieve
*A work-in-progress paper, results forthcoming?
*many interesting paper titles cited in the related work
*to read ref [4]: coding a semantic space - this seems tacked-on in this paper and it's not clear what value it brings to code the goal criterion in such a manner - wouldn't grounded theory apply here?
*glad they had a limitations section, notably identifying the scope of their definition of creativity.
!![Kandel/Heer2012] - Enterprise Data Analysis
<<cite Kandel2012>> interviewed 35 enterprise analysts from a variety of domains (contrasted with intelligence analysts). More on methodology / user profiles [[here|Information Visualization Evaluation: Qualitative Methods]]; tasks framed as challenges:
*''Discover'': data acquisition, locating data, defining database fields
*''Wrangle'': format, parse from source files, integrate multiple formats / ingesting semi-structured data, aggregation and filtering
*''Profile'': check for suitability/ data quality, make assumptions, outliers, errors. distributions
*''Model'': summarize, predict, summary stats, running regression models, performing clustering/classification, feature selection, sampling/scaling
*''Report'': communicate insights and assumptions to consumers
!!!!Comments & Questions
*Not visualization-centric, as tasks involve data acquisition, wrangling, integration; many analysts don't rely on vis
*results unsurprising
*tasks not taxonomic, but sequential and representative; each contain examples
!![Keim2002] - Information visualization and visual data mining
<<cite Keim2002>>: ''operator-based taxonomy'': dynamic projection, filtering, zooming, distortion, linking and brushing |one-dimensional, two- dimensional, multi- dimensional, text and hypertext, hierarchies and graphs, algorithms and software
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
*in <<cite Yi2007>> RW
!![Cleveland1984] - raphical perception: Theory, experimentation, and application to the development of graphical method
<<cite Cleveland1984>>
!!!!Comments & Questions
*
!![Keim2006] - Challenges in visual data analysis
<<cite Keim2006>>
!!!!Comments & Questions
*
!![Keim2010a] - Advanced visual analytics interfaces
<<cite Keim2010a>>
!!!!Comments & Questions
*
!![Kennedy/Mitchell/Barclay1996] - A framework for ~InfoVis
<<cite Kennedy1996>>
!!!!Comments & Questions
*
!![Kieras/Polson1985] - formal analysis of user complexity
<<cite Kieras1985>>
!!!!Comments & Questions
*
!![Kindlmann2014] - algebraic process for vis. design (~InfoVis '14)
<<cite Kindlmann2014>>'s algebraic process model is one about symmetries in data space, representation space, and visualization space (paper contains a succinct commutative diagram). this model is used to reason about and compare visualization design choices, and is associated with three principles:
*''representation invariance'': visualization is the same for all representations of the same data; visualizations that violate this principle are known as //hallucinators//
*''unambiguous data depiction'': for a single visualization, there is a single dataset that will map to the visualization, if multiple datasets map to the same visualization, this violation is known as a //confuser//
*''visual-data correspondence'': a change in the data should result in a corespondingly similar-in-magnitude change in the visualization space, assuming the previous two principles hold. when a change in data space is not pronounced in visualization space, the visualization has a //jumbler//, conversely, when a change in visualization space is not pronounced in data space, the visualization has a //misleader//
Their paper goes to demonstrate examples of these principles and violations, inclduing a detailed redesign case study;
They discuss data symmetries corresponding to low-level tasks, such as //operations// in <<cite Munzner2009>> or //manipulate// nodes in <<cite Brehmer2013>>.
!!!!Comments & Questions
*a systematic treatment of how to express low-level tasks as data symmetries is beyond the scope of the paper
*FW section speculates about characterizing differences in visualizations of DR data, characterizing how techniques differ
!![Kirsh/Maglio1994] - epistemic vs. pragmatic action
<<cite Kirsh1994>> explains ''epistemic actions'': gaining information about the problem at hand, vs. pragmatic actions which support users toward their goal(s).
Epistemic actions are those that make mental computation easier, faster, more reliable, less error prone, mediating intermediate results, reducing cognitive load, simplifying one's problem solving task, reducing both space and time complexity. Epistemic tasks are memory- and time-saving actions, information gathering and ''exploratory'' actions. They are intended as external checks or verifications to reduce the uncertainty of judgments. They change the input to an agent's information-processing system. Actions such as jogging memory, shatter presuppositions, hasten recognition.
>//The point of taking certain actions, therefore, is not for the effect they have on the environment as much as for the effect they have on the agent.//
They use a running example of Tetris throughout the article, in which players often make use of physical rotation and translation of "zoids", not for pragmatic goal-driven purposes, but to simplify mental computation, as physical translation and rotation is faster than mental translation and rotation (in Tetris, at least).
They claim that epistemic actions also account for distinguishing expert and novice performance, explaining rational decision making processes and preductive modelling of decisions.
!!!!Comments & Questions
*Explained by distributed cognition theory, see <<cite Pohl2012a>>'s meta-review, since epistemic actions are all about a "tighter coupling between internal and external processes", offloading computation and cognition to the environment, a cooperative and interactional relation with the world.
*Emphasis on planning behaviour of intelligent agents, control of activity, which perform poorly if only informed by pragmatic actions (and perception, attention and reasoning); intelligent agents could include epistemic action by complementing physical states with informational states: additional context for computation.
*Exploration not as a pragmatic goal but as an epistemic task; they give the example of exploring unfamiliar terrain to help decide where to camp for a night.
!![Kirsh2006] - Distributed cognition: A methodological note
<<cite Kirsh2006>>'s brief article discusses the methodological and design implications of a DC framework for HCI. As cited/summarized by <<cite Liu2008>>:
>//Formalism based abstractions on task environments or task procedures are the major approaches for studying how people interact with artifacts and other people, yet these are known to be flawed in making unreasonable assumptions about the predictability of human behaviours and characterizing the environment asa fixed set of choice points with fixed option sets rather than a place where people dynamically engage the world interactively.//
Local choice is objectively observable, but there are implicit system constraints that dictate behaviour that are less apparent. The latter refers to //pragmatic actions//, while actions that reduce computational complexity, reduce error, increase precision, and facilitate prospective memory (making environments easier to scan, notice outliers easier) are //epistemic actions//. To study the latter, participant observation, video analysis, and ethnography are necessary. This is at odds with a science of design, which require generalizations about particular mechanisms; whereas ethnography can tell us about the coordination and the interdependence of technology and practice. Similarly we strive for learnability of skills, which also involves complex interdependencies not subject to individual metrics; learning is idiosyncratic, hard to simulate.
Measuring the effectiveness of an environment or set of interdependencies in a DC system requires methodological triangulation: economic metrics, cognitive ethnography, and computational analyses. We also have to consider that people satisfice, compensate, and cope with bad design and each others' limitations; good design alone won't lead to adoption if it is a drastic shift from previous design.
On the value of simulation and modelling:
>//if we do not have living versions of different systems of coordination, how can we predict the value of re-engineering a process?//
The six assumptions of DC on interaction:
*action is closely coupled with the local environment
*thoughts and intentions are externalized
*evaluating DC systems possible with economic metrics, computational complexity, descriptive complexity, new metrics yet to be defined
*best metrics apply at many levels of analysis, from entire DC systems to individual artefacts
*coordination occurs at all levels of analysis
*history, idiosyncratic learning and collaboration, expertise, compensatory and satisficing activity matter
!!!!Comments & Questions
*Implications for studying interruptions in situ: prospective memory cues and multitasking
*Prospective epistemic tasks in coordination and organization (<<cite Heer2012a>>), brushing and linking?
!![Klahr1999] - ~Meta-Review of Scientific Discovery
<<cite Klahr1999>>'s meta-review examines scientific discovery from four complimentary approaches: historical accounts of scientific discovery, laboratory experiments (both exploratory and controlled or hypothesis-driven), observational studies of scientists in context, and computational modelling. Aside from the intrinsic value of a psychological theory of the scientific discovery process, an understanding of this process could lend itself to the design of computer programs and instrumentation for aiding scientists in their endeavors. They also posit that expert systems development research are relevant to the theory of scientific discovery and vice-versa.
The journal article covers a lot of ground, citing related work and historical examples throughout, and describes the scientific discovery process as problem solving, not fundamentally unlike all forms of human problem solving behaviour. The distinction between normal everyday problem solving and scientific problem solving is the ''combination of strong and weak methods'', the former incorporating a rich amount of ''domain expertise, methodology, and background'', while the latter is ''domain-independent, incorporating trial and error, hill climbing, means-ends analysis, planning'', and ''analogy'' - the latter of which being a bridge between strong and weak methods. With expertise, problems within one's domains are more likely to be well-defined than ill-defined, and reliance upon heuristic search-based weak methods more likely than trial-and-error.
They describe each means of studying scientific discovery with regards to several ''evaluative criteria'': face validity, construct validity, temporal resolution, likelihood of discovering new phenomena, rigour and precision of measurements, control and factorability of independent variables, external validity, and social and motivational factors. While any individual means of studying the scientific discovery process cannot fulfill all of these criteria, they can be complimentary to each other. The authors present several case studies in which a compliment of methods is used to study the discovery process. For example, important scientific discoveries by Planck, Krebs, Balmer, and Faraday are studied first by an analysis of published research, lab diaries and autobiographical accounts, then by simulating the data, hypotheses, and independent variables in exploratory lab studies, and finally using computational models given the same initial starting point.
The paper also discusses the role of surprise and ''distinguishes'' between scientific investigations that are theory or ''hypothesis driven'', and those that are ''driven by observation of an unexpected or surprising phenomena''. Studying how computational models react to surprising phenomena during an investigation and how they change course sheds light on the behaviour scientists undertake under similar circumstances.
The paper includes a discussion of the role of analogical reasoning for ''formulating initial hypotheses'', and the notion of ''multiple search spaces'', reproduced from an earlier Klahr paper from 1988. They are working towards a general theory of scientific discovery, a special case of general problem solving, one that involves parallel search of a ''hypothesis space, an experiment space, and representation space'' (abstractions, visual representations, notation), and a strategy/instrumentation space.
They conclude with the reassuring message that domain experts can be studied without domain knowledge, at the level of weak methods, problem solving and phenomena recognition processes. There is acknowledgement of the relation between problem solving and creativity, and the observation of the child-scientist: the level of creative problem-solving exhibited by children is alluring and creative, suggestive of the generality of weak-methods as-yet uninformed by domain expertise and heuristics.
!!!!Comments & Questions
*One evaluative criteria absent from this paper is the amount of preparation/overhead, particularly with respect to laboratory studies. Simulating historical discoveries in a controlled or exploratory lab setting will still require preparation of materials and tasks, and a domain expert to evaluate the results.
*Can a compliment of methods be used in the setting of VA/~InfoVis? Study of a historical discovery process (research papers, interviews, lab notes), an exploratory/phenomena-data-driven lab study to reproduce the discovery in a simulated setting, a controlled/hypothesis-driven lab study to reproduce the discovery in a simulated setting.
**Can the visual analytics process be carried out by a computational model? For a discovery process that is visually-driven, can you use an intelligent agent with computer vision to simulate the discovery process? The works they cited including computational models were numerological rather than visual data.
*While they mention the development of expert systems and computer-assisted scientific discovery in the motivation, they do not discuss how these methods have contributed to their development in this article.
*The notion of surprise (in the context of unexpected phenomena in a scientific investigation) is closer semantically to serendipity (recognition + sagacity, the ability to act on the unexpected information - <<cite Andre2009>>), than it is to insight, which can relate to both the expected and unexpected. The insight-based methodologies of <<cite Saraiya2004>>, <<cite Saraiya2006>>, <<cite North2006>>, <<cite Yi2008>>, and <<cite Saraiya2010>> all fit nicely into this notion of complementarity in investigation the scientific method, however focused on the moments of insight. Meanwhile, <<cite Mayr2010>> focuses on problem solving strategies.
!![Klein2006] - frame-based sensemaking
Two short essays (<<cite Klein2006>>, <<cite Klein2006a>>) on the ''data-frame'' model of sensemaking attempt to spell out what sensemaking is (and what it isn't).
!!!Alternative Perspectives
Sensemaking involves creativity, curiosity, comprehension, creating mental models, situation awareness, and a motivated effort to understand connections, anticipate trajectories (make predictions), and act effectively (make decisions). It serves to test and improve the plausibility of our explanations and explain apparent anomalies. It is sometimes retrospective and clarifying, while other times anticipates the future (thus facilitating prediction and decision making). It allows for the deliberation between alternative explanations, and promotes achievement of common ground.
//What it's not//: sensemaking is highly iterative, not having clear start and end points. Automated hypothesis generation and data fusion is often not helpful, forcing the user(s) to take on a passive role. It's not about trivially connecting the dots, but understanding what a dot is. It's not about keeping an open mind; it's more about evaluating alternative explanations and nascent hypotheses as they are formed, being cognizant of one's own biases.
>//"the common view of sensemaking might suffer from the tendency toward reductive explanation. What might be of help, therefore, would be a richer theory of sensemaking, one that gives shape to all the features of sensemaking listed earlier."//
!!!A Macrocognitive Model
[>img(33%, )[<<cite Klein2006>>'s data-frame sensemaking model |https://dl.dropbox.com/u/6397998/wiki/images/klein_sensemaking.png]]
This essay describes the ''data-frame theory of sensemaking'' in greater detail. Frames and data are mutually shaped by one another through means of elaboration, preservation of data within a frame (assimilation) and the questioning, rejecting, and comparing of alternative frames (accommodation). It is a closed loop between mental model formation and mental simulation. Sensemaking is a process by which multiple frames/hypotheses are considered. Maintaining multiple frames serves to avoid reductive tendencies and fixations on simple cause-effect relationships.
>//"people don't engage in simple mental operations of confirming or disconfirming a hypothesis; decision makers shift into an active mode of elaborating a competing frame once they detect the possibility that the current frame is potentially inaccurate."//
*''elaboration cycle'':
**from data, ''recognize/construct'' a frame
**with a frame, ''manage attention'', ''define, connect, filter'' the data
**''elaborate'' a frame: add and fill slots, seek and infer new data/relationships, discard data
**''question'' a frame: track anomalies, detect inconsistencies, judge plausibility, gauge data quality
**''preserve'' a frame
*''reframing cycle'':
**''reframe'': compare frames, seek new frames
!!!!Comments & Questions
*less stage-based than <<cite Pirolli2005>>; takes stance against reductive characterization of a process
*Much of the 2nd essay dedicated to design implications for intelligent agents / AI
*distinguishes ''perception'' from ''apperception'' (broader interpretation of a percept in terms of knowledge / deductive inference)
!![Knapp1995] - a task analysis approach to the visualization of geographic data
<<cite Knapp1995>>
!!!!Comments & Questions
*
!![Koedinger/Anderson1990] - abstract planning / perceptual chunks
<<cite Koedinger1990>>
!!!!Comments & Questions
*
!![Koh2011] - user-centered model for ~InfoVis
<<cite Koh2011>>
!!!!Comments & Questions
*
!![Kosara2003] - an interaction view for ~InfoVis
<<cite Kosara2003>>
!!!!Comments & Questions
*
!![Kosslyn1989] - understanding charts and graphs
<<cite Kosslyn1989>>
!!!!Comments & Questions
*
!![Kreuseler2004] - history mechanism for visual data mining
<<cite Kreuseler2004>>
!!!!Comments & Questions
*
!![Kuhn1962] - structure of scientific revolutions
In <<cite Kuhn1962>>'s //The Structure of Scientific Revolutions//, normal science is a puzzle-solving process governed by paradigms, which in turn influence a set of rules, by which the course of normal scientific experimentation, measurement, analysis, and theorizing proceed.
Paradigms are abstractions shared by those practising within a specific discipline, describing what the puzzle looks like and what parts of the puzzle remain to be solved. As a result, most of normal science is a mopping-up process in that types of problems and types of solutions are expected and arise in a routine fashion. This routine is a set of operational rules, and often these are shared across disciplines, especially in the physical sciences, while paradigms tend to be local to a particular discipline. A further commonality across disciplines is a feedback loop between experimentation and theorization.
Anomaly, invention, and discovery are violations to the order of normal science. Discovery begins with the awareness of an anomaly, a violation of the expectations that one's paradigm has provided through the course of normal science. Exploration of the anomaly follows, until the point where one's paradigm, along with its rules and theories, has shifted such that the anomaly becomes the expected. The changing of rules and theories typically and deservedly results in resistance from one's scientific community; the detection of the anomaly and the changes to one's rules relating to measurement and analysis become subject to deep scrutiny. Disagreement within the community about these discoveries lead to a state of crisis in which new theories and paradigms emerge.
At which point however, does the discovery happen? At the detection of the anomaly, or at the realization and identification of the anomaly's significance? Some examples include the discovery of oxygen, or the discovery of x-rays. I recall definitions of serendipity (<<cite Andre2009>>): a combination of detecting the unexpected result and the prior knowledge to identify and recognize the importance of the unexpected result, which seems to indicate that a dilettante or a layman could make a discovery, however the significance of this discovery could not be realized without the necessary prior knowledge.
!!!!Comments & Questions
*See <<cite Andre2009>> on serendipitous discovery, <<cite Klahr1999>> on scientific discovery
*<<cite Case2008>> points out that Kuhn uses the term "paradigm" in a handful of different ways, leading to confusion.
!![Lam2008] - interaction cost framework
<<cite Lam2008a>> surveyed a ridiculous number (484) of ~InfoVis papers, from which 32 report on usability issues, resulting in a list of 61 problems and 14 design recommendations. Lam places these into a framework of ~InfoVis interaction costs that is modelled on Norman's //Seven Stages of Action// framework, covering both high and low-level user tasks, which includes the ''gulf of execution'' (low-level, physical motion costs - mapping user intent to allowable system actions) and the ''gulf of evaluation'' (high-level, cognitive interpretation costs - mapping system state to expectations/intents). Lam adds the ''gulf of formation'' to represent high-level cognitive decision costs related to data analysis and intent formation.
There are 7 costs in the framework:
*gulf of ''formation'' (high-level)
**costs to form goals: question formulation, choosing a data subset / focus, information foraging, tracking information scent, choosing interface options to match goals
*gulf of ''execution'' (low-level)
**//translating questions into system operation//: choosing among sequences of operations
**//choosing among multiple input/device operations//: resolving mode errors, understanding modes of use, acknowledging mode changes
**//physical motion costs//: reducing travel, adjusting target size; dragging, moving
*gulf of ''evaluation'' (high-level)
**//visual cluttering costs (difficulty interpreting system state)//: resolving occlusion
**//view-change costs (interpreting changes in visual perception)//: perceiving changes, understanding intelligent interface / augmented interface behaviour, keep objects in memory, multiple view interpretation, maintaining local/global associations
**//state-change costs on evaluating an interpretation//: comparing/contrasting between different projections, refinding, reflecting, understand changes
!!!!Comments & Questions
*A lot of papers and reports from IJHCS, ~InfoVis, TOCHI (but not CHI), VAST, ~EuroVis; relied on keyword search to narrow down literature base
*[[full report list|http://www.cs.ubc.ca/~hllam/res_icost.htm]]
*low-level interaction focus, ignored papers that focus solely on high-level user goals
*"less than 30% of user studies of interactive interfaces mentioned interaction"
!![Lammarsch2012] - extended task framework for temporal exploratory data analysis
<<cite Lammarsch2012>>'s ~EuroVA workshop paper extend the top-down task characterization for spatial and temporal data of <<cite Andrienko2006>>, as a response to the top-down / general formulation of these tasks, by developing a formal language for specifying low-level / bottom-up temporal EDA tasks based on the characteristics of time defined by <<cite Aigner2011>>: ''scale'', scope, arrangement, ''viewpoints'' (ordered vs, branching time), ''granularities'', time primitives, and determinancy (bold terms emphasized in the text). They claim a rule set that is a complete and formal task framework.
They are motivated by the incompleteness of prior task taxonomies, particularly for temporal data (<<cite Amar2005>>, <<cite MacEachren1995>>, <<cite Peuquet1994>>, <<cite Andrienko2006>>.)
!!!!Comments & Questions
*their formal language does not explicitly address multiple time viewpoints (branching time); they have to be specified separately?
*cited by R2:
>//A weakness of the related work is that the "what" part is greatly underrepresented. This is intentional, but I disagree about the decision. There should be a least examplary publications about this, for example starting with the task by data type taxonomy by Shneiderman which is already mentions as [56] in Section 2.3, but does only provide a minor "how" answer in Chapter 3. From there, examples for each basic datatype are possible. For time, this could be// <<cite Lammarsch2012>>
*Does not connect how provides a formal grammar for connecting to why is not directly addressed.
!![Lee2006] - Graph Vis Taxonomy (BELIV)
<<cite Lee2006>> extends the <<cite Amar2005>> low-level taxonomy to network visualization tasks, a task-by-data type taxonomy that expands upon <<cite Shneiderman1996>>'s treatment of network graph tasks. Their tasks are hierarchical, wijh <<cite Amar2005>>'s low-level primitives at the lowest level of the task hierarchy. They include a section on high-level tasks, suggesting that the network tasks are mid-level tasks.
Their low level tasks includes <<cite Amar2005>>'s list: ''retrieve value, filter, compute derived value, find extremum, sort, determine range, characterize distribution, find anomalies, cluster, correlate''. They extend this with 3 new low-level tasks: ''find adjacent node''; ''set operation'' (perform operations on a set of nodes - e.g. find the intersection of a set of nodes); ''scan'' (differs from ''retrieve value'', as a value need not be retrieved, but many items are reviewed);
Their network graph taxonomy is described as follows:
*''topology tasks''
**''determine adjacency'': find [node], find adjacent [nodes], retrieve value [on nodes] | filter | compute derived value (count) | find extremum
**''determine accessibility'': find, find adjacent, retrieve value | count | filter, retrieve value
**''find common connection'': find, find adjacent, find (again), find adjacent, set operation (intersect)
**''determine connectivity''
***''find shortest path'': find, find adjacent (breadth-first search)
***''find clusters'': scan, count
***''find connected components'': scan, count
***''find bridges'': scan, find bridge
***''find articulation points'': scan, find articulation point
*''attribute tasks'': subsumes topology tasks and adds ''filter, compute, range, distribution'' in addition to count tasks
**''find node attributes'': filter, retrieve value | count, scan
**''find link attributes'': find, find adjacent, filter | find extremum, retrieve value
*''browsing tasks''
**''follow path'': find, find adjacent, scan (repeat) (from nodes A-B-C-D)
**''revisit'': scan, find adjacent (repeat)
*''overview tasks'': compound exploration task; estimate network size, determine existence of clusters and connected components; find patterns and outliers
They also describe several high-level tasks not accounted for in this taxonomy, that is they are presently disjoint from this taxonomy: compare/contrast multiple networks, node/link identity resolution, data cleaning/wrangling, giving a group/cluster a meaningful name, determine how a network changes over time.
!!!!Comments & Questions
*Tree task list used as starting point: [[InfoVis 2003 contest|http://cs.umd.edu/hcil/iv03contest]]
*[[network task wiki|http://www.infovis-wiki.net/index.php/Tasks_Taxonomy_for_Graphs]]
**more potential tasks:
***[[what is the general structure of this graph?|http://www.networkweaving.com/blog/2006/09/nola-networks.html]]
***[[what is the distribution of vertex degree in this graph? (That is, how are well-linked nodes different from under-linked nodes?)|http://research.microsoft.com/research/pubs/view.aspx?type=Publication&id=1601]]
***[[how many As are linked to Bs? How many As link to other As?|http://www.crookedtimber.org/2005/05/25/cross-ideological-conversations-among-bloggers/]]
*Table 2 compares several network visualization techniques/tools w.r.t. their ability to perform the tasks described by the taxonomy, a comparative evaluation in itself
*overview (4.4) and high-level tasks (5): how to separate? are these compound mid-level tasks? no explicit linking from mid-level tasks to high-level tasks is attempted.
*overview tasks w.r.t. clusters and components are already given as topology tasks?
*what's an operation? a low-level task?
!![Lee2012] - beyond mouse and keyboard (~InfoVIs 12)
<<cite Lee2012a>> argues that traditional WIMP ~InfoVis tools are drowning in functionality, ignoring developments in user-centred interaction research in mainstream HCI, and is stuck in a bubble of WIMP-interface task- and data-centric interaction research (or fixated on the process of authoring visualizations according to the visualization pipeline).
>//"these new types of "natural" interaction seem to to bypassed the field of ~InfoVis"//
In their characterization, an interaction begins with an intent, followed by an action, which triggers some feedback from the system and in return a reaction to that feedback. WIMP interfaces are inefficient, and direct manipulation (or post-direct manipulation) interfaces allow people to concentrate on their tasks without concentrating on the interaction/interface; thereby minimizing the cognitive distance b/w intent and execution.
>//"~InfoVis interaction taxonomies consider the system reaction or the resulting functionality part of the interaction timeline largely discuss interaction from the viewpoint of interaction tasks to be performed with and on the data, or from interaction techniques that can be performed on the data or its visual representation. we take a human-centric approach instead and do not specify data or tasks"//
They discuss new interaction research from HCI and implications for ~InfoVis from the perspective of the individual use, the technology, the sociality, and the interplay between these, and not with regards to the data or task.
!!!!Comments & Questions
*Provocative. However, interaction; the paper sidesteps the issue of understanding how mainstream ~InfoVis users (professionals analyzing data using WIMP interfaces) carry out their tasks; embodied and proxemic interaction are interesting, flashy topics currently popular in mainstream HCI, but these are often associated with casual, not professional contexts.
*An exception is that of collaborative interfaces used in cooperative decision making. Nevertheless, while studying natural, situated interactions in the context of collaborators is important, there is still a need for studying this phenomena at the level of abstract tasks: what are the collaborators working toward?
*How can you discuss tasks without discussing the user? For WIMP interfaces, do you need to discuss the embodied state of the user? No.
*Vis pipeline addendum (Spence (2007 text), Carpendale ~PhD thesis): presentation transformation (is this really an addendum?)
*By ignoring data and task they ignore the top half of the nested model
*They correctly point out that extant ~InfoVis taxonomies of interaction techniques for data exploration "are often mixed and discussed interchangeably with those related to an analyst's tasks with a visualization"
!![Li2009] - information search tasks - LIBR
<<cite Li2009>>
!!!!Comments & Questions
*
!![Lipford2010] - Helping users recall their reasoning process
<<cite Lipford2010>>
!!!!Comments & Questions
*
!![Liu,Stasko2008] - Distributed cognition as a theoretical framework for ~InfoVis
<<cite Liu2008>> discusses distributed cognition (DC), not as a testable theory in itself, but as a framework or model for describing the interaction and coordination of humans, visualizations, and interaction techniques. Rather than thinking of visualizations as scaffolds or artefacts that //amplify cognition// occurring solely in the mind of the user, information visualizations are external representations that are part of a larger cognitive system, emphasizing the "embodied, enculturated, situated in local interactions, and distributed or stretched across humans and artefacts".
>//Cognition is an emergent property of the interactions between an individual and the environment through perception and action rather than a property bounded inside an individual.//
Information is propagated as a series of ''representation states'', some are external within or between representations and artefacts, while some are internal to individuals (or shared between individuals). These states are coordinated and are interdependent; this is what DC theory aims to characterize. Traditional cognitive science focuses solely on the internal representations and abstract pattern-action rules. Interaction with external artefacts in various states is indicative of internal transformations; the challenge lies in studying those internal representations and matching them to the external states. [[Think Aloud Protocol]] can be useful, yet is unnatural. External representations speed up problem solving (see <<cite Zhang1994>> reference).
For understanding interaction, we can observe interactions with external representations but the coordination with internal states is missing, these are various and subtle (<<cite Kirsh2006>>); another way to describe these are by ''pragmatic'' goal-directed actions and ''epistemic'' mental coordination actions (<<cite Kirsh1994>>). An issue with the design and evaluation of interactive software is that only the former are acknowledged and afforded in the design, epistemic and appropriated actions / affordances are often lost.
>//the design of new digital artefacts risks destroying many of the most valuable aspects of current ways of doing things which we do not yet fully understand// - summarizing Hutchins' //Cognition in the Wild//
DC also accounts for collaboration between actors using multiple tools, performing pragmatic and epistemic actions in unison, coordinating and resolving internal and external representations.
Regarding the process or product of ''insight'', coordination and conceptual change can help explain when and how insight "happens".
DC has implications for how we evaluate ~InfoVis tools. As a framework, it can help us to think about studying situated cognition, acknowledging both external and internal states of information, and capturing the process of coupling, coordination, interaction strategies for sensemaking and analytical reasoning, the creation and evolution of external representations, and the role of interaction in aiding understanding. ''Cognitive ethnography'' (<<cite Hollan2000>>, Hutchins' //Cognition in the Wild//) suggests an interpretive, field method approach to evaluation, which can in turn contribute to the design of more focused and controlled lab studies.
>//A science of interaction should not be just a taxonomy of interaction techniques or a framework of the abstracted task procedures; it should be a scientific approach to understand how cognition emerges as a property of interaction between external and internal representations.//
Taxonomies are bottom-up, while a DC framework can serve as a top-down lens
!!!!Comments & Questions
*DC's approach to insight not yet well established; it's a particular type of coordination between internal and external?
*Collaboration and DC similarly not very developed in this paper
*The case against bottom-up taxonomies?
*Suggests studying individual domains to find/develop a general framework
*DC cannot account for perception
*DC's limitations in prescription and prediction <<cite Bederson2003>>
!!!!Further reading:
*//The Craft of Information Visualization: Readings and Reflections//
**Bederson, B. and Shneiderman, B. //Theories for understanding information visualization// (2003)
*Gholson, B. and Shadish, W. R. and Neimayer , R. A. and Houts, A.C. (eds.). [[Psychology of Science: Contributions to Metascience|http://www.amazon.ca/Psychology-Science-Contributions-Barry-Gholson/dp/0521203201/ref=sr_1_1?s=books&ie=UTF8&qid=1355268129&sr=1-1]] (2011)
**Tweney, R. D. //A framework for the cognitive psychology of science//
*Flack J. and Hancock P. and Cairn J. and Vincente, K. (eds.). [[Global Perspectives on the Ecology of Human-Machine Systems|http://www.amazon.ca/Global-Perspectives-Ecology-Human-Machine-Systems/dp/0805813810/ref=sr_1_1?s=books&ie=UTF8&qid=1355268371&sr=1-1]] (1995)
**Woods, D. D. //Toward a theoretical base for representation design in the computer medium: Ecological perception and aiding human cognition.//
!![Liu2010] - Mental models, visual reasoning and interaction in information visualization: a top-down perspective
<<cite Liu2010>> build upon <<cite Liu2008>>'s DC paper and continue with a consideration of mental models and the interplay between internal and external visualization. Some definitions of mental models in HCI / cognitive science are helpful, while others aren't actionable, hence the lack of interest in mental models in the vis. community. They remark that cognitive science's focus on internal visualization is an ironic parallel with the visualization research community's focus on external visualization - external visualizations can not entirely, nor should they, replace or substitute mental models. Mental models, coupled with an understanding of visual reasoning and the study of user interaction can help viusaliztion designers.
>//"a focus on either mental models or external visualizations tends to margnilize interacion, a core phenonmenon yet to be understood"// (proceeds to cite <<cite Yi2007>>)
Users internalize visualizations and engage in ''mental simulation''. Mental models must be functional, rather than mental maps of data, for this reason. Mental maps are an overly restrictive definition for mental models. In visualization, mental models need not be maps at the data item level (though they can serve this purpose if necessary), since this is externalized, but rather functional modesls of how the visualization system works and how one can interact with it in parallel to their own visual reasoning.
Dynamics between mental models and external visualizations include ''internalization'' of functional models, ''processing'', ''augmentation'' (the central part of a DC system), and ''creation'' (discovery and innovation), includes defintions of discovery by analogy and simulative modeling, in addition to logical inference and deduction.
We ''construct'' and ''manipulate'' mental models in working memory for reasoning, anticipation, and planning. How does interaction with external visualizations map to these activities? Interaction can be physical or imply a change in the state of the visualization, but this definition fails to account for all possible human actions in interacting with visualizations (cites Tufte's //Visual Explanations// p.50), both passive and active - an example of passive and active navigation and differences in modality. This means that there can be many ways (passive and active interaction) of accomplishing a task, and affordances and constraints of external visualizations reduce the number of ways.
Interaction serves 3 major functions/intentions (which in practice feedback into one another in a cyclic manner), and is further broken down into types of action that go beyond the limits of interactive features (too numerous to taxonomize in the paper):
*''external anchoring''
**''project'': projection is involved in mental siumulation, finding a context, contextual anchors such as axis ticks and text labels, giving meaning to the external visualization
**''locate'': refers to locating external anchors for internal-external coupling
*''information foraging'': <<cite Pirolli2005>>, <<cite Card1999>>
*''restructure'': subsumes <<cite Yi2007>>'s definition of ''reconfigure'' and ''encode''
*''explore'': refers to a lack of information on hand for hypothesis forming, forces semi-random behaviour, exploring the environment to find useful information - coordination of eye movements and zooming/panning
*''cognitive offloading'' (subsumes ''highlight'', ''arrangement'' of data to fit mental model)
**''create'': creating material anchors, reducing load on WM, dynamic pointers
**''save/load'':
They continue to discuss implications for design, evaluation, and theory.
For design, they refer to the gap between designers' and users' mental models (but that users shouldn't be assigned the duty of visualization construction due to their lack of expertise in programming and visual design: let designers ''construct'', users ''manipulate''), and the utility of relying on ''foundational schemas'', such as physical embodiment and orientational metaphors (which is why Shneiderman's mantra <<cite Shneiderman1996>> is so effective), particularly when designing "for the masses". They mention phsyical constraints of interaction and how this may hamper mental simulation.
For evaluation, current methods, be they time+error or insight-based methods, are too focused on outcomes, not on process, nor do they consider individual differences. Observations, interpretations, and the think-aloud protocol, despite its interference in ongoing tasks, is nevertheless valuable for understanding process, especially in longitudinal studies.
For theory, they suggest the need for a "taxonomy of mental simulation" by which we can use to inform the design of interaction, though not all mental model simulation should be offloaded.
!!!!Comments & Questions
*cite for future eval. work.
*a partial task/action taxonomy
*interaction as a partial window to mental simualtion / mental model internalization, processing, augmentation, and creation. interaction is both passive and active <<cite Spence2007>>.
!![Lohse1990] - classifying visual knowledge representations
<<cite Lohse1990>>
!!!!Comments & Questions
*
!![Lohse1993] - classifying visual knowledge representations
Lohse, G. L. (1993). A cognitive model for understanding graphical perception. //Human-Computer Interaction, 8//, 353-388.
There are three types of graphical perception tasks: ''point reading'', ''making comparisons'', ''identifying trends''.
!!!!Comments & Questions
*cited by <<cite Friel2001>>, corroborates with their task classification.
!![Lohse1994] - A classification of visual representations
<<cite Lohse1994>>
!!!!Comments & Questions
*
!![MacEachren1994] - visualization in modern cartography
[>img(33%, )[<<cite MacEachren1994>>'s cartography research agenda |https://dl.dropbox.com/u/6397998/wiki/images/maceachren1994.png]]
<<cite MacEachren1994>>'s cartographic research agenda describes a research space for cartography and visualization:
*''tasks'': presenting knowns vs. revealing unknowns
*''users'': public vs. private
*''interaction'': high vs. low
*''visualization and communication'': exploration, confirmation, synthesis, presentation
!!!!Comments & Questions
*(book unavailable)
*slide from Roth RE. [[The science and practice of cartographic interaction|http://www.slideshare.net/reroth/the-science-and-practice-of-cartographic-interaction-13336159]]. In: GeoInformatics 2012. Hong Kong: June 15th.
!![MacEachren1995] - //How Maps Work: Representation Visualization, and Design// (text)
<<cite MacEachren1995>> defines a set of low-level tasks centred around the questions/interests of the user with respect to:
*''existence of data element''
*''temporal location''
*''time interval''
*''temporal pattern''
*''rate of change''
*''sequence''
*''synchronization''
!!!!Comments & Questions
*cited and discussed in <<cite Aigner2011a>> w.r.t. identification and localization tasks
!![MacEachren1999] - interactivity and spatiotemporal data
<<cite MacEachren1999>>: ''objective-based taxonomy'': identify, compare, interpret; ''operator-based taxonomy'': assignment, brushing, focusing, colormap manipulation, viewpoint manipulation, sequencing
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
!![Marchionini2006] - Searching vs. Browsing
<<cite Marchionini2006>>'s communications of the ACM article gives a high-level summary of the differences between lookup and exploratory search, the latter subsuming learning and investigating tasks. The gist of it is in Figure 1:
*''Lookup'': fact retrieval, known item search, navigation, transaction, verification, question answering
*''Exploratory Search / Learn'': knowledge acquisition, comprehension / interpretation, comparison, aggregation / integration, socialize
*''Exploratory Search / Investigate'': accretion, analysis, exclusion / negation, synthesis, evaluation, discovery, planning / forecasting, transformation
Some case studies /examples follow (Open Video Digital Library video browsing and Relation Browser faceted database search). Discusses the new costs associated with exploring (as opposed to lookup): exploratory search offers more opportunities for diversions and advertising on the web.
!![Mackinlay1986] - Automating the design of graphical presentations of relational information
<<cite Mackinlay1986>>
!!!!Comments & Questions
*
!![Maletic2002] - A task oriented view of software visualization
<<cite Maletic2002>>
!!!!Comments & Questions
*
!![Marr1979] - A computational theory of human stereo vision
<<cite Marr1979>>
!!!!Comments & Questions
*referred to by RR in thesis proposal defence
!![Marr1982] - Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (book)
<<cite Marr1982>>
!!!!Comments & Questions
*referred to by RR in thesis proposal defence: why / what / how
!![Mayr2010] - Characterizing Problem Solving
<<cite Mayr2010>>'s BELIV '10 paper argues that analysis of representative users' creative problem solving techniques sheds more light on the exploratory visual analytics process than [[Insight-Based Evaluation]], and is thus a superior evaluation technique. They define types of problems (well-defined (one solution), ill-defined (exploratory)), as well as problem solving strategies (schema-driven, search-based). They encourage visualization designers to accommodate both types of strategies.
''Methodology'': real-world data sets of varying levels of real-world complexity, two visualization systems. Interaction was logged, tracked viewing behaviour, and used the [[Think Aloud Protocol]]. 12 participants had to solve a set of well-defined and ill-defined problems. Tasks included:
*//reading the data// (locating/extracting data, well-defined),
*//reading between the data// (relationships, interpolation, well-defined), and
*//reading beyond the data// (extrapolation, ill-defined).
''Results'': strategies were highly variable within and between subjects for tasks and tools, but some data sets suggested certain strategies (an interaction of data set and strategy). A correlation between strategy selected and solution quality was found, but nevertheless multiple strategies should be afforded in interfaces. Their approach was easy to apply and replicate.
!!!!Comments & Questions
*a subsequent 2011 IV journal paper (<<cite Mayr2011>>) covers the same study in greater detail
*Their methodology and relations to other ongoing projects were unclear.
*It wasn't stated how their problem sets (both well-defined or ill-defined) were selected or who generated them (and how much time this took relative to the analysis of their results).
*I wasn't buying their conclusion that analyzing problem solving techniques answers more than [[Insight-Based Evaluation]]. They do not relate solution quality to insights (they assume that the problem solving process leads to insight generation).
**They could have done a head-to-head comparison of the two techniques (albeit limited being in a lab setting with artificial problem sets, users, and data sets), with dependent variables being time to analyze results vs. time to design predefined problems, amount of expert / domain knowledge required, and when expertise is needed.
*They admit that problem-solving subprocesses are still unclear, despite a wealth of literature on low-level and perceptual-level task taxonomies relating to problem solving, insight generation, information foraging.
*Their method did not compare head-to-head against the [[Insight-Based Evaluation]], so claims about which is more effective are questionable.
*Did not appreciate writing style. I will forego reading IV journal paper, but a related IEEE journal paper re: participatory design looks like it may be relevant.
!![~McNamara2012] - BELIV on mental model formation in information foraging
<<cite Mcnamara2012>>
!!!!BELIV notes:
*//General Specific//: analytics (general) and analysis (analysis)
*analytics (general): methods, problem classes, extensibility, defined domains
*analysis (specific): methods instantiated in tool, application to particular domain, individual differences, domains evolve
*18 intelligence analysts: interviews, observations of analysts using a text analytics tool for LSA, LDA
*comparing the VA tool against card-sorting exercise (incredible amount of variation)
*role of expertise that gets cut out of LSA/LDA algorithms
*role of information scent, information foraging
*impact of working memory, cognitive load, adaptive/custom searches
*explanatory power, larger than the phenomenon, occurring across fields
*Martin Heidegger: ready at hand vs. present at hand? tools under evaluation are more often present to hand (objective analysis of the tool) rather than ready to hand (is the task supported?)
!!!!Comments & Questions
*
!![Meister1985] - information needs of scientists
<<cite Meister1985>>'s classification of task behaviours from his text //Behavioural Analysis and Measurement Methods//:
*''perceptual processes''
**''searching for and receiving information'': detect, inspect, observe, read, receive, scan, survey
**''identifying objects, actions, events'': discriminate, identify, locate
*''mediational processes''
**''information processing'': categorize, calculate, code, compute, interpolate, itemize, tabulate, translate
**''problem solving and decision making'': analyze, calculate, choose, compare, compute, estimate, plan
*''communication processes'': advise, answer, communicate, direct, indicate, inform, instruct, request, transmit
*''motor processes''
**''simple/discrete'': activate, close, connect, disconnect, join, move, press, set
**''complex/continuous'': adjust, align, regulate, synchronize, track
!!!!Comments & Questions
*cited by <<cite Vicente1999>> (Table 3.1)
*very similar in content to <<cite Mullins1993>>
!![Menzel1964] - information needs of scientists
<<cite Menzel1964>>
!!!!Comments & Questions
*
!![Merkelbach/Schraagen1994] - A framework for the analysis of cognitive tasks
<<cite Merkelbach1994>>
!!!!Comments & Questions
*
!![Mirel2014] - Scientists' sense making when hypothesizing about disease mechanisms from expression data and their needs for visualization support
<<cite Mirel2014>>' //BMC Bioinformatics// article adopts <<cite Brehmer2013>>'s task typology as a way to summarize tasks
!!!!Comments & Questions
*
!![Morse2000] - evaluating vis. tools w/ the Zhou/Feiner1998 Taxonomy
<<cite Morse2000>> use the <<cite Zhou1998>> taxonomy to evaluate 4 different visualization tools for information retrieval. They translate a subset of the task taxonomy into abstract domain tasks, and again into concrete tasks particular to the datasets used in the study. They do not use the entire taxonomy due to time constraints, they restrict the set of tasks to those whose parameter lists varied significantly, and to sample broadly rather than deeply. The 10 tasks corresponded to: ''compare, associate, distinguish, rank, cluster, correlate, locate, categorize, identify, compare''. They omitted ''identity, encode, background, emphasize, reveal, generalize, switch'' and ''identity, distribution'' from <<cite Wehrend1990>>.
195 subjects participated in either a 2-term or 3-term IR condition. They experienced 4 visualization types in a random order: table, text, spring, icon (stacked bar). Dependent metrics were time and error for each task and interface, subjective preferences for each interface.
Tasks performance differed (both time/error) significantly between the 2-term and 3-term IR condition, with the exception of the ''distinguish'' and ''rank'' tasks. Regardless of IR term-count condition and visualization type, ''Associate'', ''identify'', and ''rank'' were associated with high correctness scores. ''Cluster'', ''locate'', and ''compare'' were error-prone and took significantly longer. Some tasks interacted with visualization type: a word list (the base case) was faster for ''rank'', ''cluster'', ''correlate'' and ''locate'', than the other visualization types (table, stacked bar, spring). The table was slowest overall. Subjects' subjective preferences matched the visualization types that they objectively performed best on.
!!!!Comments & Questions
*Visualizations selected were in information retrieval domain - some taxonomy tasks may not generalize to other domains?
*Prescribed tasks :( student/novice participants, no domain knowledge; they admit some task instructions could be misleading/ambiguous.
*Did not validate the <<cite Zhou1998>> taxonomy, nor did they select/design tasks according to presentation intents, visual accomplishments/implications or by the inform/enable dichotomy
*Their overall results section states that visual tasks vary in difficulty; are they of the same granularity?
*There was also no correlation between task parameter number and task difficulty; how was this measured? Why did they have this hypothesis? They measured task difficulty by amount of time required to complete the task; different tasks took different amounts of time; why do they have the assumption that difficulty and time are related?
*They admit that not all IR tasks need to be visual tasks (hence the inclusion of a text list as one of the visualization types).
*writing not very good, many typos.
!![Meyer2012] - nested model revisited (BELIV '12)
[>img(100%, )[nested model revisited|http://www.cs.ubc.ca/labs/imager/tr/2012/NestedModelExt/figures/blocks.png
]]
<<cite Meyer2012>>:
>//"knowledge of tasks is deeply incomplete."//
>
>//"we lack task blocks at the middle level. There is a pressing need to propose and validate mid-level task taxonomies that bridge the gap between finding the minimum value of an attribute and con- firming or refuting a hypothesis"//
>
>//"this lack of both blocks and mappings at least partially accounts for the difficulty of going from a problem characterization to an abstraction when conducting a design study."//
!!!!BELIV notes
*nested model blocks and guidelines: mappings and comparisons
*what are the actionable guidelines for between-level mappings
!!!Discussion:
*''EB'': what community are we addressing?
**''MG'': our community is too self-serving; we need to be more actionable to
*''Tim Lebo'' (RPI): are the blocks formalized? the recipe book for visualization?
*''Q'': what are the users' tasks? their contexts?
*''Q'': we need to know more about the low-level and high-level composite tasks?
**''Tamara Munzner'': we need a better understanding of mid-level tasks;
**''Laura ~McNamara'': auditor audience evaluation vs. stakeholder evaluation; no mid-level tasks, these are constantly evolving; we should be empowering users to determine these abstractions themselves rather than catalogue themselves
**''Sheelagh Carpendale'': danger of guidelines, formulaic writing; studies never give you anything: precision, generalization, realism; we as a community don't look to realism in studies;
**''MG'': evaluations that are persuasive and realistic are hard; both persuasive and actionable is hard; it won't occur through a single study, but through multiple means
**''Heidi Lam'': if we don't aspire to transferability, we'll never be done, we'll keep evaluating tools/techniques in single contexts without transferable actionability to other contexts
*''John Stasko'': most evaluations are neither persuasive nor actionable, but aimed at refinement, improving one's tool/technique; formative design, included to please reviewers
*''TM'': confirm, refine, reject, propose (DSM); persuade / make actionable - what are the terms? what do we do?
**''EB'': is it necessarily bad to be non-prescriptive or make vacuous assertions?
**''MG'': what about the term "inspire" (is this guidance?)
**''HL'': framework and transferable context?
**''TM'': guidelines helping or hurting?
**''SC'': humans are complex adaptive systems and visualizations are linear systems; we're trying to apply linear methods without acknowledging the full complexity; you can't re-assemble complex adaptive systems
**value to non-actionable evaluation, as it may not be actionable //now//, but research contexts change over time
!!!!Comments & Questions
*
!![Mullins/Treu1991] - Task protocol specification (TPS)
<<cite Mullins1991>>
!!!!Comments & Questions
*
!![Mullins/Treu1993] - Network Interaction Task Taxonomy
<<cite Mullins1993>> compiled a task list of 143 tasks from the user interface task taxonomy literature to date, grouped them around a cognitive model that incorporated a hierarchy of higher-level "internal" cognitive tasks (the root of the hierarchy) and lower-level "external" basic tasks (user input/output, visual object manipulation).
They define a ''task'' as "//a logically consistent unit of work that brings the user closer to the accomplishment of some goal / set of related activities.//"
[>img(40%, )[Mullins and Treu's task-based cognitive model of a user information processing system (Fig. 3)|http://cs.ubc.ca/~brehmer/research/mullins_task.png]]
Their task taxonomy works better as a directed graph than as a hierarchical taxonomy, mirroring their cognitive model of information processing.
Beginning with the highest-level cognitive tasks and working outwards, their hierarchical task taxonomy is as follows (inner nodes are low-level, outer-nodes are higher-level):
*//mediation and coordination//: the root node for all cognitive tasks; not always supported at the interface level, these take place primarily as cognitive tasks without interface interaction
**''assess'': low-level comparison against an expected value (or range of values):
***''success'': compare results with user goal image
***''evaluate'': determine value, amount, worth of an entity according to some metric
***''test'': decide upon integrity of a signal/message/mechanism
***''compare'': discover similarities/differences b/w multiple entities
***''verify'': confirm a value
**''analyze'': organize information into abstractions:
***''interpret'': translate symbolic context into a reference/meaning
***''calculate'': reckon, compute, mathematical/logical reduction/deduction,
***''categorize'': classify/sort data/information/entities according to some classification scheme
***''count'': identify entities and increment magnitude
***''itemize'': list/specify all components of a grouping
***''tabulate'': tally/enumerate frequencies/values of list or table
**''synthesize'': information processing tasks (closely related to analyze):
***''integrate'': pull together, mentally organize, taxonomize
***''translate'': convert/change from one representational scheme to another via a mapping
***''remember'': retain/recall information for later consideration
***''prioritize'': order information according to priority
***''estimate'': mentally judge, approximate, gauge according to incomplete data, draw conclusion
***''extrapolate'': assign approximate value to future point based on preceding points
***''interpolate'': assign approximate value to interim point based on bracketing reference points
**''solve'' problems / plan:
***''formulate'': put ideas together, integrate a plan
***''plan'': devise a contingency, anticipate possible outcomes
***''program'': create formal description of a plan
***''diagnose'': determine cause of condition, run tests, check signs/symptoms
***''decide'': arrive at answer/choice/decision/conclusion
***''choose'': select among alternatives
**''learn'': gain a better understanding of aspects of the system:
***''query'' in depth: pursue help information at increasing levels of detail
***''tutorial'': use intelligent tutoring service
***''browse'': explore with not other purpose that to discover available features
**other cognitive meta-tasks listed for convenience:
***''undo'': cancel last action (''redo'' implicit?)
***''reset'': return to default, purge old context for new context
***''cross-reference'': lookup of related information according to indexing scheme (i.e. read manual)
*//object space modelling//: objects in the system are modelled by users
**''create'' objector parts of objects:
***''associate'' objects: create references among objects
****''name'': give object a title
****''group'': connect items by association to treat as single items
****''link'': create directed link b/w items
***''assemble'' objects:
****''aggregate'': combine multiple components into new composite entity
****''paste'': copy an object into new location
****''overlay'': (subsumed by ''aggregate'') but retain separability
****''insert'': insert one entity into another
***''replicate'' objects:
****''copy'': duplicate an entity
****''instance'': reproduce an original entity from a master entity, retain inheritance
****''store'': save entities to secondary storage
***''introduce'' objects or data
****''data entry'': manually add new data or acquire through ''monitor''
****''object definition'': create a new object by defining it
****''restore'': copy from permanent storage
**''eliminate'' objector parts of objects:
***''remove'' object(s) or parts of objects:
****''cut'': remove entity from one place, but hold in buffer
****''delete'': remove and destroy entity
****''purge'': remove unwanted entities from storage according to a some rule
***''disassociate objects'':
****''rename'': change entity's name
****''ungroup'': (subsumed by ''segregate'') opposite of group, treat aggregated objects as segregated objects again
****''unlink'': (subsumed by ''segregate'') remove directed link between objects
***''disassemble objects'':
****''segregate'': partition aggregated objects
****''filter'': (subsumed by ''screen''): selectively eliminate layers of overlaid composite
****''suppress'': (subsumed by ''screen''): conceal (affects appearance only)
****''withdraw'': remove embedded object from its parent
**''activate'' entities (process control):
***''execute'' a process:
****''start'': run from initial state
****''invoke'': start after adjusting context of process
***''change status'':
****''re-start'': restore process back to running status
****''foreground/background'' switch: or vice-versa, implies an interim suspend
***''stop process'':
****''suspend'': stop and hold in abeyance (suspend)
****''terminate'': stop active process without regard for restartability
****''set-aside'': store contents of work area in some buffer to execute without access to the user
****''quit'': exit interactive session
**''indicate'' entities:
***''pick'': opt/choose an entity/group of entities
***''reference'': opt/choose an entity/group of entities by their symbolic name
***''mark'': indicate a place/entity for later use, assign symbolic name
**other object manipulations:
***''edit'': modify definition of some object, arrange information
***''display'': arrange object definitions into prescribed format
*//communicate//: concerned with the semantics of meaning of information, connecting the user with the outside world
**''transmit'' (syntactic output to physical interface of external system via motor functions (interactions)): send or pass on information/data, or request it
***''call'': send signal to recipient that message is forthcoming
***''acknowledge'': confirm message received
***''respond'': answer a request for information
***''suggest'': (subsumed by ''inform'') offer a suggestion for consideration
***''direct'': provide explicit instructions
***''inform'': pass on / relay new knowledge/data
***''instruct'': (subsumed by ''inform''): teach/educate/provide remedial data
***''request'': solicit, query, ask for information
***''form-message'': put together message into correct syntactic form
***''record'': document something, log
***the preceding transmit tasks are accomplished via ''user output'': direct communication between the user and system:
****''object manipulation'': modify/change entities or attributes modelled (displayed?) by the user interface:
*****''transform'': manipulate / change entity's attribute
*****''stretch'': distort/change a shape
*****''sketch'': free-hand sketching
*****''re-orient'': (subsumed by ''orient'')
*****''shape'': (subsumed by ''stretch'')
*****''pan'': show parts of view outside of display bounds
*****''zoom'': show more/less detail
*****''move'': move an item/entity into a new location
****''application interactions'': articulate a message, physically communicate with the system:
*****''select'': choose among alternatives via some command
*****''position'': indicate a position/set of coordinates
*****''orient'' an entity
*****''path'': generate a path / series of positions
*****''quantify'': (subsumed by ''select''): specify a numeric value
*****''discrete value'': (subsumed by ''select''): choose among a constrained set of values
*****''text'': input a text string
*****''hold'': continuous control
*****''push/pull'': apply directional force
*****''control'': change direction, rate, magnitude of physical force
**''reach through'': find a communication protocol acceptable to all parties, a negotiation
**''receive'' (syntactic input from physical interface of external system via motor functions (interactions)): get, acquire, obtain an incoming message (even when not requested)
***''attend'': get expected/need information
***''monitor'': get ancillary/non-critical information
***''notice'': get unexpected yet important information
***''filter'': filter out inconsistent/unimportant information
***''accept'': (subsumed by ''monitor''): receive unsolicited general background information, not of current interest
***the preceding ''receive'' tasks are accomplished with the ''perceive'' system input task:
****''acquire'': search for and receive information
*****''observe'': attend to presence / status of an entity
*****''scan'': glance quickly, look for patterns / anomalies, general impression, monitor several channels
*****''search'': purposeful exploration / looking for specific items or those that match criteria
*****''inspect'': examine carefully, view w/ critical appraisal
*****''extract'': directed, attentive reading/observing/listening/gleaning the meaning
*****''screen'': (ignore): filter out noise, select input channel, stripping redundancies/irrelevant information
*****''detect'': discover/notice (unsolicited) occurrence - awareness of presence/absence of entities
****''identify information'': message characterization by type or source
*****''discriminate'': roughly classify an entity by set/group membership on basis of partial information
*****''recognize'' specific information of an entity
*****''identify'': recognize nature of an object according to implicit characteristics (e.g. file icon = file)
*****''locate'': seek out / determine location/place of an entity without explicit plan for doing so
They evaluated their task list with a small questionnaire study involving 7 network analysts. This reduced the task count from 143 to 120. They went on to design a network analysis prototype system with these tasks in mind, however about a dozen tasks were subsumed by others.
!!!!Comments & Questions
*Comprehensive and flexible task list, almost too broad, well-beyond network visualization, even beyond all visualizations; covers tasks for a broad range of user interfaces.
*Task taxonomy quite large, lots of cross-cutting and redundancies and tasks subsuming others (occasional loops)
*task taxonomy often used ''entities, objects'', and ''items'' interchangeably, as it does for ''events, activities'', and ''processes''
*full task taxonomy given as an appendix, though it was the main contribution
*according to their ''task'' definition, what is an ''activity''? A sub-task?
*2-way communication between system and user, emphasis on internal cognitive activities occurring in the user's cognitive model
!![Munzner2009] - nested model
[>img(40%, )[Munzner's nested model|http://www.cs.ubc.ca/labs/imager/tr/2009/NestedModel/figs/nested.png
]]
<<cite Munzner2009>>:
>//"the reductionist assumption that complex sensemaking tasks can be broken down into low-level components is unlikely to be completely true."//
!![Munzner2009a] - vis. chapter
<<cite Munzner2009a>>'s Fundamentals of Graphics book chapter touches on tasks:
Citing the nested model paper (<<cite Munzner2009>>): the need for task characterization, "different questions, or tasks, require very different visual encodings."
>//"In most cases, users know they need to somehow view their data but cannot directly articulate their needs as clear-cut tasks in terms of operations on data types"//
In the abstraction phase, domain tasks are mapped to generic tasks (<<cite Amar2005>>) and data-type-specific tasks (<<cite Shneiderman1996>>). Simultaneously, the data is simplified, transformed and derived, each a task in themselves - deciding //what// to show.
Regarding interaction, Munzner mentions <<cite Shneiderman1996>>'s mantra, as well as a need for low-latency feedback, selection with visual indication / highlighting. Her mention of common interaction techniques is change-centric:
*''Navigation'' changes the viewport
*''Sorting'' changes the spatial ordering
*changing the visual encoding
Data reduction is treated separately. Tasks referred to include overview and aggregation, filtering and navigation/zooming, (focus+context, not a task but a technique).
!!!!Comments & Questions
*Discusses at length the costs of interactivity, animation - what role does costs play in the taxonomy?
!![Munzner2014] - Visualization Analysis and Design
<<cite Munzner2014>> extends <<cite Brehmer2013>>'s typology as follows:
*''why'' @@color:#bb0000;(actions)@@:
**@@color:#bb0000;''analyze''@@
***''consume'': //discover// (generate / refine / verify hypotheses), //present//, //enjoy//
****''search'': //lookup// (identity and location of target known), //locate// (only identity known), //browse// (only location known), //explore// (neither is known)
****''query'': //identify// (one target), //compare// (multiple targets), //summarize// (all targets)
***''produce'': @@color:#bb0000; //annotate//, //record//, //derive//@@
*@@color:#bb0000; ''why'' (targets)@@:
**@@color:#bb0000; ''all data'': //trends//, //outliers//, //features//@@
**@@color:#bb0000; ''attributes'': //one// (value, extremes, distributions), //many// (similarities, correlations, dependencies)@@
**@@color:#bb0000; ''network data'': //paths//, //topologies//@@
**@@color:#bb0000; ''spatial data'': //shape//@@
*''what'': //inputs// and //outputs//
**@@color:#bb0000; ''tables'': //items//, //attributes//@@
**@@color:#bb0000; ''trees and networks'': //attributes//, //links//, //items/nodes//@@
**@@color:#bb0000; ''fields'': //attributes//, //grids//, //positions//@@
**@@color:#bb0000; ''geometry'': //items//, //positions//@@
**@@color:#bb0000; ''clusters, sets, lists'': //items//@@
*''how'':
**''encode'': @@color:#bb0000; //arrange (express, separate, order, align, use)//, //map// (color (hue, luminance, saturation, transparency), (position, size, angle, curvature), (region, shape), motion (direction, rate, frequency))@@
**''manipulate'': //select//, //navigate//, @@color:#bb0000;--//arrange//--@@, //change//, @@color:#bb0000;--//aggregate//--@@, @@color:#bb0000;--//filter//--@@
**@@color:#bb0000; ''facet'': //juxtapose//, //partition//, //superimpose//@@
**@@color:#bb0000; ''reduce'': //filter//, //aggregate//, //embed//@@
**@@color:#bb0000; --''introduce'': //import//, //record//, //derive//, //annotate//--@@
!!!!Comments & Questions
*
!![Nersessian2006] - cognitive-cultural systems of the research laboratory
<<cite Nersessian2006>>
!!!!Comments & Questions
*
!![Nersessian2008] - Mental modeling in conceptual change
<<cite Nersessian2008>>
!!!!Comments & Questions
*
!![Norman1988] - design/psychology of everyday things
<<cite Norman1988>>'s seven stages of action model (ch. 2, pp. 45-53), an "approximate model", wherein the stages aren't in reality this discrete, and can be started at any point, reflective of the observation that some goals are opportunistic while others are planned:
*goals:
**1. ''forming the goal'', something to be achieved, often imprecisely/vaguely specified for everyday tasks, sometimes goals are opportunistic rather than planned
*gulf of execution:
**2. ''forming the intention'', specific statements of what is to be done, linking goals to actions (sequences of action, internal commands)
**3. ''specifying the action'', based on visible affordances of the system
**4. ''executing the action'', physical interaction
*gulf of evaluation:
**5. ''perceiving the state of the system'', our perception of the world
**6. ''interpreting the state of the system''
**7. ''evaluating the outcome'': does the outcome match our goal?
!!!!Comments & Questions
*the basis of <<cite Roth2012>>'s objective (stage 2) / operator (stage 3) / operand (stage 4 - 5) classification of extant interaction taxonomies
*re: goals > intentions > sequences of action: where are tasks? are tasks the combination of these, the translation of goals into action via intentions? are actions described by objectives / operators / operands? is //action// synonymous with //interaction//?
!![Oliveira2003] - From visual data exploration to visual data mining
<<cite DeOliveira2003>>
!!!!Comments & Questions
*
!![Patterson2001] - CTA for intelligence analysis
<<cite Patterson2001>>
!!!!Comments & Questions
*
!![Peuquet1994] - it's about time
<<cite Peuquet1994>>: "It's about time: A conceptual framework for the representation of temporal dynamics in geographic information systems" - describes the triad model of spatio-temporal data and possible questions pertaining to //when, what, and where//. Given two of these, a common question is to determine the remaining third.
!!!!Comments & Questions
*cited by <<cite Lammarsch2012>>, <<cite Roth2012>>, <<cite Andrienko2006>>
!![Plaisant/Carr/Shneiderman1995] - Image-browser taxonomy
<<cite Plaisant1995>> presents a taxonomy about browser interfaces, both single-view and multiple-view variants, including overview and detail displays. Most of the article is spent describing the primitives and attributes of browser displays, categorizing existing browser types. Within this, a small, high-level task taxonomy justifies the existence of different browser types:
*''image generation'': most of time spent on detail view (e.g. CAD systems)
*(open-ended) ''exploration'': unknown context / environment, incomplete information, details-on-demand (example: exploring a city map, adventure game)
*''diagnostic'' (a special case of exploration): examine boundaries / differences at micro/macro level (example: tissue visualization in pathology; VLSI circuit inspection)
*''navigation'': environment is known, overview only required for context; path following, comparison of alternatives, requesting details or related information in another view on-demand (example: wayfinding)
*''monitoring'': focus on anomalies while maintaining a "big picture" overview (example: network / factory production integrity)
!!!!Comments & Questions
*Not a particularly helpful task taxonomy, but nice categorization of single and multiple view displays
*Subtasks implied from examples; despite domain ties in task taxonomy, I don't think these domain ties are necessary
*The browser taxonomy implies some low-level tasks: zooming (and adjusting zoom parameters), panning, window management: positioning and resizing; article does not describe explicit ties from these low-level tasks to high-level task taxonomy, only to certain browser types.
!![Pike/Stasko/Chang2009] - The science of interaction
[>img(50%, )[<<cite Pike2009>>'s analytic discourse |https://dl.dropbox.com/u/6397998/wiki/images/pike2009.png]]
<<cite Pike2009>> addresses interaction, an overloaded term. There is low-level interaction between a user and an interface, and high-level interaction between a user and a problem space. The article follows from <<cite Liu2008>> in that interaction at a high-level can be described by a ''distributed cognition'' / situated cognition framework in that interaction as a means of conceptual manipulation is supported by computational tools, but does not take place exclusively within them, nor exclusively within the minds of users. The user and the environment become collaborators in the discovery process.
>//"The interaction is the inquiry."//
''Dialogical inquiry'' between user and environment resists adhering to rules and cannot therefore be evaluated in a strict quantitative fashion. It encourages open exploration.
>//"In many analysis tasks, goals are unstable, and a straightforward progression down a path of discovery is impossible. A breakdown in analytic discourse is not necessarily bad; in fact, it is often under conditions of breakdown that new discoveries are made."//
Pike and colleagues characterize interaction at two levels, that of the low level of interface interactions (interactions described by <<cite Amar2005>>, <<cite Jankun-Kelly2007>>: change representation, uncover patterns, relationships, trends, other features) and at the higher level of information space interactions (tasks described by <<cite Yi2007>>), which involves interaction with information on hand, prior knowledge, ones colleagues and environment. They describe the process of analytic discourse, a relationship between user tasks and goals at high and low levels. They too distinguish between ''techniques'' and ''intents'', in that the former should never be considered as an ends in itself, but "a means to support the user's information understanding".
There is mutual feedback between users' goals and tasks and interactive visualization:
*user ''goals'' and ''tasks''
**//high-level//: ''explore, analyze, browse, assimilate, triage, assess, understand, compare''
**//low-level//: (following task taxonomy of <<cite Amar2005>>, also influenced by interaction history/provenance work of <<cite Jankun-Kelly2007>>, <<cite Shrinivasan2008>>): ''retrieve value, filter, sort, compute derived value, find extremum, correlate, determine range, cluster, characterize distribution, find anomalies''
*interactive visualization
**//high-level//
***//representation intents//: ''depict, differentiate, identify, show outliers, compare''
***//interaction intents//: (following task taxonomy of <<cite Yi2007>>): ''select, explore, reconfigure, encode, abstract/elaborate, filter/connect''
**//low-level//
***//representation techniques//: (operands) charts, graphs, networks, treemaps, parallel coordinates
***//interaction techniques//: ''selection, brushing, dynamic query, pan/zoom,''
They state that despite many extant task/interaction taxonomies, they see a need to understand a relationship between task/interaction components and modes of inquiry: ''abduction'' (constructing hypotheses), ''deduction'' (refuting prior hypotheses), ''induction'' (verification, ranking alternative hypotheses, identifying the best explanation). This will allow for the development of "systems that are able to recognize, reflect, and support the generation of insight by their users".
The remainder of the article highlights challenges in interaction foreseen in the VA field over the next 5 years (2009-2014), an agenda for future research: ubiquitous embodied interaction, capturing user intentionality, knowledge-based interfaces, collaboration, principles of design and perception, interoperability for integrated interaction, and evaluating the costs and benefits of interaction.
!!!!Comments & Questions
*We lumped <<cite Yi2007>> as being low or low-mid, along with <<cite Amar2005>>, their list of "low-level interaction" includes "uncover patterns/trends", which we see as being mid-level
*mapping from high-to-low still incomplete, mapping from "high-level" to actual high-level: abduction, deduction, induction still needed
*mapping from representations to interactions, intents to techniques similarly incomplete
!![Pirolli1995] - Information foraging in information access environments
<<cite Pirolli1995>>
!!!!Comments & Questions
*
!![Pirolli2005] - Sensemaking
[>img(40%, )[Pirolli's sensemaking loop for intelligence analysis, derived from cognitive task analysis|http://dydan.rutgers.edu/PDDALab/dev/images/flow.png]]
<<cite Pirolli2005>> conducted a [[Cognitive Task Analysis]] and the [[Think Aloud Protocol]] to better understand the information foraging and sensemaking task-flow of intelligence analysts, providing a descriptive view of the process for the purpose of designing future technology.
At a high-level, the sensemaking process as described here is not at all different from the processes expert meteorologists engage in described by <<cite Trafton2000>>: getting a high-level overview, building a qualitative mental model, refining and adjusting the model, and extracting quantitative and qualitative information from that model in the form of decisions or a written brief (i.e. a weather forecast, flight plan). The figure at right captures these smaller sub-processes. <<cite Pirolli2005>> does however break down the tasks at a finer level, distinguishing between top-down and bottom-up processes, invoked in a "opportunistic mix".
The [[Cognitive Task Analysis]] results suggest several leverage points in the ''information foraging'' process in which technology may be able to improve efficiency or quality of the results. Each of these points are associated with costs, and therefore can be rephrased as heuristics when designing/evaluating tools to support the process:
*determine the exploration-enrichment-exploitation tradeoff - time and space allotted to monitoring, narrowing, and reading/analyzing. [[Focus+Context]] techniques serve as a compromise in this signal/noise tradeoff, one in which the costs of analyzing too many documents in detail (false positives) may be worth less than missing relevant documents (false negatives) - this cost differential is domain specific
*facilitate scanning, recognizing, selecting items for further attention - highlighting (i.e. pre-attentive codings)
*allow shifting attentional control - allowing bottom-up vs. top-down processing
*facilitate follow-up searches
In the ''sensemaking loop'', the [[Cognitive Task Analysis]] results suggest additional leverage points (focused around generating and managing hypotheses, reasoning and decision making), again rephrased as design/evaluation heuristics:
*use external working memory for analysts to manage evidence and hypotheses
*support adequate comparison of alternative hypotheses
*provide clear confirmation or dis-confirmation of hypotheses
!!!!Comments & Questions
*While the focus of this article is on intelligence analysts, I expect the processes depicted may generalize to a number of research fields involving domain expertise, schematic knowledge structures, pattern recognition and anomaly detection (the latter triggering the sensemaking loop).
*Missing references / references not numbered: where is Bodnar (2003)?
*Were these later rephrased as design/evaluation heuristics? Applied to other domains?
*Apparently there is another related paper regarding intelligence analysis in greater detail, or with a different focus - however web searches all point to this paper.
!![Pohl2010] - Log file analysis of EDA
<<cite Pohl2010>>'s BELIV workshop paper presents findings from a study in which they evaluated use of a visualization tool intended for exploratory data analysis by means of qualitative and quantitative log file analysis. They discuss the role and characterization of EDA from the perspective of perceptual psychology and Gestalt psychologists: "//insight into the structure of a problem and by productive restructuring of the problem//".
Their study involved 32 computer science students performing 2 exploratory data analysis tasks designed by domain collaborators. Insight characterization was also performed alongside collaborators. Logs recorded their interface-level interactions with the tool. They performed qualitative and quantitative analysis of the log file. They found task-specific usage patterns, trends of macro-tasks (common sequences of interactions). They characterize some of these patterns according to Gestalt definitions of problem solving.
!!!!Comments & Questions
*summary: "we expected EDA to be random behaviour, but it wasn't! (mostly)"
*non-native English writers, hard to read at times; visualization tools referred to as methodologies or methods interchangeably.
*They do not address the insights, whether or not they were achieved by users, whether collaborators commented on their utility, or how different strategies allowed users to arrive at insight
*Student (non-domain) participants, fixed time limit on study, prescribed open-ended tasks, but prescribed nevertheless
*is EDA problem solving? an unstated assumption
*No full task-taxonomy is created, however they make references to several pre-existing task taxonomies without fully committing to them.
*Their hypotheses are based on what? Odd mixing of perspectives, a post-positivist perspective not really appropriate.
*"//subjects prefer to interact with the data and do not experiment with the visualization options//" - and "//users choose their preferred method of visualization early on and do not experiment with these options very much//" what, beyond the interface in question, are we to take out of these statements?
*interaction patterns may not result in insight - what to they result in?
*why did they believe that "//subjects would follow no particular pattern and use visualization methods randomly//"?
*one-off interactions not fitting in larger macro-task patterns were "//considered to be random user activities and were ignored//"
*quantitative analysis superfluous and didn't tell us anything that qualitative analysis didn't already; what are they referring to as "significant"?
*figure 4 is a mystery
!![Pohl2012] - ~VisWeek '12 Interaction Theories and VA
[>img(50%, )[<<cite Pohl2012a>>'s characterization of theories based on relevant categories for VA (Fig. 1)|https://dl.dropbox.com/u/6397998/wiki/images/pohl2012a.png]]
<<cite Pohl2012a>> reviews theories to explain visual analysis using VA/Vis tools. Using these tools is a means of problem solving; often exploratory complex tasks where there is no clear solution/defined path. The theories include:
**''sensemaking'' <<cite Pirolli2005>>, <<cite Klein2006>>, <<cite Klein2006a>>: an emphasis on what happens in analysts' mind, a decision making process; background knowledge plays a crucial role (the schematizing step); sensemaking represented as stages: acquire information (searching and filtering, extraction and fusion of information from different sources), (generate representations), make sense of information, create something new (generate hypotheses), and act on it (<<cite Attfield2010>>); disadvantages: doesn't explain interaction or perception, a rigid assumptions of stages (in reality these are very much intertwined); frame-based model of <<cite Klein2006>>, <<cite Klein2006a>> may be superior to <<cite Pirolli2005>>, as Klein et al. explains cycles of elaboration and reframing, explains collaborative sensemaking, no rigidly defined stages.
**''gestalt theory'': a holistic and iterative problem restructuring; distinguishes routine and non-routine problems; related to insight (no common definition, but usually involves non-routine problems that involve conceptual change, creative thinking; is insight a product or a process? (<<cite Chang2009>>, <<cite North2006>>); advantages: emphasizes the visual representational aspect of problem solving, that representations must change and transform, thereby overcoming the problem of fixation; gestalt theory doesn't explain iterative reasoning processes in steps or problem solving strategies (''MB'': really?)
**''distributed cognition'' <<cite Hollan2000>>, <<cite Liu2008>>: critical role of artefacts and interactions with artefacts, affordances provided by the design of the artefact, however the process of interaction is still not well understood; the artefact is scaffolding; DC takes collaboration into account; DC also explains //epistemic actions// (<<cite Kirsh1994>>: gaining information about the problem at hand, vs. //pragmatic actions// which support users toward their goal(s); disadvantages: doesn't explain interaction (''MB'': really?), or address visualization in particular;
**''graph comprehension'': <<cite Friel2001>>: reading the data, reading between the data, reading beyond the data; <<cite Kosslyn1989>>: 3 levels of comprehension: syntactic, semantic, and pragmatic levels of processing; interaction and interpretation, bottom-up and top-down processes / expectations and prior knowledge; advantages: task taxonomies could adopt language of graph comprehension theory, transformations based on data type and their attributes; doesn't account for the roles of prior expertise/knowledge (''MB'': really?), skills; disadvantages: doesn't account for interaction, interacting with complex visualizations not well understood; different graph comprehension theories don't integrate well
**''skill-rule knowledge theories'': emphasis on errors, and hasn't been applied in VA / ~InfoVis, so we don't know if it can explain dynamic interaction; a mid-level of abstraction between graph comprehension (low-level) and sensemaking (high-level); a cyclic skill-rule-knowledge loop: processing level (learned skills, schematic, highly automatic), rule-based level (diagnosis, aim to minimize mismatches between known rules and current situation, application of heuristics), knowledge level (classical reasoning, problem solving, use of analogies, abstract analysis): explains problem solving in known/unknown circumstances, shifting from knowledge to skill levels; not specific to visual data exploration, emphasis on errors rather than success criteria, not focused on open-ended exploration
The discussion section compares theories, establishes a schema for comparison, reviews their strengths and weaknesses: (labelled as high and low, see adjacent Figure)
The take home message: high-level theories don't explain TASKS, they don't give us design guidelines: we don't have cohesion, we need to extend theory, merge, and develop new theories.
!!!!Comments & Questions (from ~VisWeek '12)
*"A warning that this presentation will contain almost no pictures", but this is still relevant for VA.
*begins with a definition of VA.
*Is characterization sufficient for VA/ Vis - what's missing?
*high-level tasks: cite for task paper
*a meta-analysis of theories to explain VA
*no citations for latter 4 theories (in talk)
*hard-to-read comparison plot the only visual in the slide deck
*I was hoping more out of the comparison
!!!!Questions
*'''Bill Ribarsky'': re: interaction strategies, what about it? Which theory emphasizes reasoning through interaction?
**distributed cognition (DC) comes closest, but more is needed
*''Bill Ribarsky'': but does DC handle conversation b/w human and machine?
**it explains the communication with the data, perhaps not the tool?
*''Q'' what about Klein's data-frame sensemaking model, that does emphasize interaction: connect, refine, elaborate?
*''Enrico Bertini''? did you reorder columns and rows in the theory comparison plot to find patterns? (laughter from audience)
**No. ''Brian Fisher'': a ~VisWeek talk with one vis in and it gets criticized.
!!!!Comments & Questions (from reading the paper)
*Gestalt theory not well defined? no guidelines from Gestalt theory?!?! It doesn't explain iterative processes? What about restructuring and overcoming fixation?
*Distributed cognition: it's about interacting with artifacts, but we don't understand the interaction? It explains affordances but not interaction?
*I disagree that graph comprehension theory doesn't account for prior knowledge - what about top-down processes, semantic level of processing?
!!!!Further reading:
It turns out that there's little agreement at level of high-level models; books cited by <<cite Pohl2012a>> (Proc. VAST '12):
*Carroll, J.M. (ed.) (2003). [[HCI Models, Theories and Frameworks|http://www.amazon.com/HCI-Models-Theories-Frameworks-Multidisciplinary/dp/1558608087/ref=sr_1_1?ie=UTF8&qid=1354062239&sr=8-1&keywords=HCI+Models%2C+Theories+and+Frameworks]]. Morgan Kaufmann
**Perry, M. //Distributed cognition//
**Pirolli. P. //Exploring and finding information// (sensemaking)
*Davidson, J. E. and Sternberg, J. R. (eds.) (2003). [[The Psychology of Problem Solving|http://www.amazon.com/Psychology-Problem-Solving-Janet-Davidson/dp/0521797411/ref=sr_1_1?s=books&ie=UTF8&qid=1354062265&sr=1-1&keywords=The+Psychology+of+Problem+Solving]]. Cambridge University Press
**Davidson, J. E. //Insights about insightful problem solving// (gestalt theory)
**Pretz, J. E. and Naples, J. //Recognizing, defining, and representing problems// (gestalt theory)
*Freedle, R. (ed.) (1990). [[Artificial Intelligence and the Future of Testing|http://www.amazon.com/Artificial-Intelligence-Future-Testing-Freedle/dp/0805801170/ref=sr_1_1?s=books&ie=UTF8&qid=1354062285&sr=1-1&keywords=Artificial+Intelligence+and+the+Future+of+Testing]]. Lawrence Erlbaum
**Pinker, S. //A theory of graph comprehension//
*Holyoak, K. J. and Morrison, R. G. (eds.) (2005, 2007). [[The Cambridge Handbook of Thinking and Reasoning|http://www.amazon.com/Cambridge-Handbook-Reasoning-Handbooks-Psychology/dp/0521531012/ref=sr_1_1?s=books&ie=UTF8&qid=1354062310&sr=1-1&keywords=The+Cambridge+Handbook+of+Thinking+and+Reasoning]]. Cambridge University Press
**Novick, L. R. and Bassok, M. //Problem solving// (gestalt theory)
**Tversky, B. //Visuospatial reasoning// (graph comprehension)
*Hutchins, E. (1995). [[Cognition In The Wild|http://www.amazon.com/Cognition-Bradford-Books-Edwin-Hutchins/dp/0262581469/ref=sr_1_1?s=books&ie=UTF8&qid=1354062328&sr=1-1&keywords=Cognition+In+The+Wild]]. The MIT Press
*Mirel, B. (2004). [[Interaction Design for Complex Problem Solving|http://www.amazon.com/Interaction-Design-Complex-Problem-Solving/dp/1558608311/ref=sr_1_1?s=books&ie=UTF8&qid=1354062352&sr=1-1&keywords=Interaction+Design+for+Complex+Problem+Solving]]. Elsevier, Morgan Kaufmann
*Shah, P. and Miyake, A. (eds.) (2005). [[The Cambridge Handbook of Visuospatial Thinking|http://www.amazon.com/Cambridge-Handbook-Visuospatial-Handbooks-Psychology/dp/0521807107/ref=sr_1_1?s=books&ie=UTF8&qid=1354062611&sr=1-1&keywords=The+Cambridge+Handbook+of+Visuospatial+Thinking]]. Cambridge University Press
*Reason, J. (1990). [[Human Error|http://www.amazon.com/Human-Error-James-Reason/dp/0521314194/ref=sr_1_1?ie=UTF8&qid=1354065753&sr=8-1&keywords=reason+human+error]]. Cambridge University Press
*Rogers, Y. and Rutherford, A. and Bibby, P.A. (eds.) (1992). [[Models in the Mind: Theory Perspective and Application|http://www.amazon.com/Models-Mind-Perspective-Application-Computers/dp/0125929706/ref=sr_1_1?s=books&ie=UTF8&qid=1354062368&sr=1-1&keywords=Models+in+the+Mind%3A+Theory+Perspective+and+Application]]. Academic Press
**O'Malley C. and Draper S. //Representation and interaction: Are mental models all in the mind// (distributed cognition)
*Salomon, G. (ed.) (1993). [[Distributed Cognition: Psychological and Educational Considerations|http://www.amazon.com/Distributed-Cognitions-Psychological-Considerations-Computational/dp/0521574234/ref=sr_1_1?s=books&ie=UTF8&qid=1354062389&sr=1-1&keywords=Distributed+Cognition%3A+Psychological+and+Educational+Considerations]]. Cambridge University Press
*Sternberg, R. J. and Davidson, J. E. (eds.) (1995). [[The Nature of Insight|http://www.amazon.com/Nature-Insight-Bradford-Books/dp/0262691876/ref=sr_1_1?s=books&ie=UTF8&qid=1354062415&sr=1-1&keywords=The+Nature+of+Insight]]. The MIT Press
**Dominowski, R. L., and Dallob, P. //Insights and problem solving// (gestalt theory)
**Weisberg, R.W. //Prolegomena to theories of insight in problem solving: A taxonomy of problems// (gestalt theory)
**Mayer, M. //The search for insight: Grappling with gestalt psychology's unanswered questions//
!![Pohl2012] - Analysing interactivity in information visualisation
<<cite Pohl2012b>>
!!!!Comments & Questions
*
!![Pousman2007] - casual ~InfoVis
<<cite Pousman2007>>
!!!!Comments & Questions
*
!![Pretorius2014] - Tasks for multivariate network analysis
<<cite Pretorius2014>>
!!!!Comments & Questions
*
!![Purchase2008] - Theoretical foundations of information visualization
<<cite Purchase2008>>
!!!!Comments & Questions
*
!![Quinn2011] - human computation
<<cite Quinn2011>>
!!!!Comments & Questions
*
!![Raskin2000] - //the Humane Interface//
<<cite Raskin2000>>'s taxonomy of ''elementary actions'' (p.104) for nearly every interface include those performed in various combinations. He states that the actual number of elementary actions are quite limited, especially with 2D mouse and keyboard input. Content can be:
*''indicated'': point at
*''selected'': distinguish from other content
*''activated'': clicked on
*''modified'': or ''used'' by being:
**''generated'': from non-empty to empty
**''deleted'': from non-empty to empty
**''moved'': interted in one place and deleted from another place
**''transformed'': changed to another data type
**''copied'': sent / recieved from an external device or duplicated at a different internal location (e.g. email, print, stored on disk, copied into another document.)
Raskin goes on to discuss the additional operation of ''query'', or to bring up further information or options on the item, but also as "operation on one object that brings into view another, related object".
On tasks vs. operations:
>//That the interfaces of all applications arise from a small set of elementary operations confirms that the applications themselves, as rich and varied as they are from a task-oriented point of view, are not all that different from one another from an interface-oriented point of view. This fundamental similarity can be exploited to create powerful computer systems of unprecidented simplicity and productivity.//
Methods for choosing and marking content:
*''highlighting'': adding a recognizable distinction to an object, allowing one to ''determine'' by passive observation that the object has a special status
*''indicating'': distinguished from passive ''highlighting'' and a click-indusced ''selection'', ''indicating'' is associated with mouse-over
*''selecting'': explicitly creating a special status for an object via clicking, dragging, lassoing (single or composite selection)
!!!!Comments & Questions
*cited by R3 (secondary):
>//What I further miss in the context of the 'manipulate' and 'introduce' tasks is "deletion". In this sense, it might also be useful to have a look at// <<cite Raskin2000>> (p. 104)
*overlap with ''how''
*low-level operations are few, high-level tasks are many combinations of low-level operations?
*defintion of ''selection'' not internally consistent (does it mean 'distinguish' or 'highlight' or actively clicking on an object?)
!![Rasmussen1985] - Decision support in supervisory control
<<cite Rasmussen1985>>
!!!!Comments & Questions
*
!![Rensink2012] - science of visualization
<<cite Rensink2012>> discusses prospects for a science of visualization with regards to low-level visual perception tasks, borrowing experimental methodologies from vision science but retaining stimuli and visual tasks from the visualization domain: perception of correlation, pattern detection, cluster detection, outlier detection, grouping, finding convex hull of points, following a curve, finding point of maximum intensity, etc.
He proposes the ''extended vision thesis'' and the ''optimal reduction thesis''. The former refers to the viewer and visualization system as a single system. The latter refers to reducing a task to low-level operations in the extended systems - these are low-level visual tasks and do not refer to interactivity.
>//"the existence of a systematic framework for visualization is problematic neither in principle nor in practice"//
!!!!Comments & Questions
*Metrics: accuracy, variability, time required
*Low-level visual perception tasks; interactive tasks not addressed
*High-level objective tasks not addressed (other than "find something interesting" - large gap between visual task to detect correlation and "find something interesting")
*emphasis on static visualization tasks, likening to basic visual tasks in vision science research, whereas interactive tasks are analogous to higher-level visual perception, such as scene perception
!![Rheingans2002] - "Are we there yet?" exploring visualization
<<cite Rheingans2002>>
!!!!Comments & Questions
*
!![Ribarsky2009] - Science of analytical reasoning
<<cite Ribarsky2009>>
>//In representing the sensemaking model, there are three primary classes of objects: Stages, Artifacts and Data Tasks sensemaking stages and artifacts are represented as classes, and the relationships between them can be encoded in the class properties. This will produce a set of hierarchical relationships that shows the sensemaking tasks at the highest layer of abstraction, the artifacts at the next lower layer and the data tasks at the lowest layer. // (p.260)
what follows is about inferring tasks from recorded interface interactions, more automated history and analytical provenance stuff.
!!!!Comments & Questions
*
!![Robertson1991] - A methodology for choosing data representations
<<cite Robertson1991>>
!!!!Comments & Questions
*
!![Rogowitz/Treinish1993] - Architecture for rule-based Vis
<<cite Rogowitz1993>>
!!!!Comments & Questions
*
!![Roth1990] - Data Characterization for Intelligent Graphics Presentations
<<cite Roth1990>> characterize domain-independent user goals to compliment their data characterization; they are broken down into two groups, one relating to the function of a display (<<cite Chi1998>>'s view space) and the distribution/relatedness of the data (value space; filtering and deriving data values). In the display function space, several low level tasks are characterized:
*''value lookup''
*''compare within a relation''
*''compare across/between relations''
*''determine distribution(s)''
*''determine correlation(s)''
*''indexing/sorting''
!!!!Comments & Questions
*User goals doesn't get much attention (article mostly about visual encoding and and data abstraction)
*significant overlap with <<cite Wehrend1990>> (see below)
*not very informative for tasks/goals
!![Roth1991] - Automating the presentation of information
<<cite Roth1991>>
!!!!Comments & Questions
*
!![Roth2011] - ~GeoVis interaction (dissertation)
<<cite Roth2011>>
!!!!Comments & Questions
*
!![Roth2012] - ~GeoVis cartographic interaction primitives / meta-analysis of Vis task taxonomies
<<cite Roth2012>>'s Cartographic Journal article addresses the question of //how// interaction should be afforded in geographical/cartographic applications, the article refers a survey of extant task taxonomies, both in and outside of geographic/cartographic/spatial visualization domains, distinguishing taxonomies into three areas according to <<cite Norman1988>>'s //Design of Everyday Things// classification: user objective, operator (tools/widgets), and operand (data objects/abstractions).
>"//A particularly difficult part of the problem is to develop a typology of geospatial interface tasks//". - Cartwright et al (2001).
Roth's intention was not to develop a new taxonomy of interaction primitives, but to organize extant taxonomies into a conceptual framework (although he later states that this allows for the development of a composite theoretical-empirical taxonomy), to examine concordances and discordances. The purposes of this is to develop a common lexicon for describing interactions and competing interface designs (<<cite Beaudouin-Lafon2004>>'s descriptive power), to inform the design of experiments (evaluative power?), and to inform the design and evaluation of interactive maps (generative power?). This conceptual framework will allow designers to "go beyond the simple reporting of these new designs and additionally provide evidence as to why they work through administration of interaction experiments informed by the framework".
Roth's summary of extant task (interaction) taxonomies is reproduced here; italicized citations are not currently described elsewhere in this annotated bibliography. They are sorted into categories based on <<cite Norman1988>>'s Seven stages of Action model, wherein ''objective''-based taxonomies are placed at the ''forming the intention'' stage, ''operator''-based taxonomies are placed at the ''specifying an action'' stage - made aware via interface affordances, and ''operand''-based taxonomies are placed between the ''executing the action'' stage and the ''perceiving the state of the system'' stage, between execution and evaluation (the ''world'').
| !Citation | !Objective-based | !Operator-based | !Operand-based |
| ''<<cite Amar2005>>'' |retrieve value, filter, compute derived value, find extremum, sort, determine range, characterize distribution, find anomalies, cluster, correlate | | |
| ''<<cite Andrienko2003>>'' |identify, compare | |space, time, objects |
| ''//Becker & Cleveland (1987, brushing ~SPs)//'' | |identification, deletion, linking, brushing, scaling, rotation, dynamic parameter control| |
| ''//Becker, Cleveland, & Wilks (1987, dynamic graphics)//'' | |highlight, shadow highlight, delete, label | |
| ''//Blok et al. (1999, ~GeoVis lit.)//'' |identify, compare | | |
| ''<<cite Buja1996>>'' | |focusing, linking, arranging views | |
| ''<<cite Chi1998>>, <<cite Chi2000>>'' | | |data, analytical abstraction, visualization abstraction, view |
| ''<<cite Chuah1996>>'' | |encode data (graphic), set- graphical-value (graphic), manipulate objects (graphic), create (set), delete (set), summarize (set), join (set), add (data), delete (data), summarize (data), join (data)|graphical, data, control |
| ''<<cite Crampton2002>>'' |examine, compare, (re)order/(re)sort, extract/suppress, cause/effect | |data, representation, temporal dimension, contextualizing interaction |
| ''<<cite Dix1998>>'' | |highlight and focus, accessing extra information, overview and context, same representation-changing parameters, same data- changing representation, linking representations | |
| ''//Dykes (1997, ~GeoVis lit.)//'' | |observer motion, object rotation, dynamic comparison, dynamic re-expression, brushing | |
| ''<<cite Edsall2008>>'' | |zooming, panning/re- centering, re-projecting, accessing exact data, focusing, altering representation type, altering symbolization, posing queries, toggling visibility, brushing and linking, conditioning | |
| ''//Haber & ~McNabb (1990, ~SciVis lit.)//'' | | |data, derived data, visualization abstraction, view|
| ''<<cite Keim2002>>'' | |dynamic projection, filtering, zooming, distortion, linking and brushing |one-dimensional, two- dimensional, multi- dimensional, text and hypertext, hierarchies and graphs, algorithms and software |
| ''<<cite MacEachren1999>>'' |identify, compare, interpret |assignment, brushing, focusing, colormap manipulation, viewpoint manipulation, sequencing | |
| ''//Masters & Edsall (2000, Vis toolkit lit.)//'' | |assignment, brushing, focusing, colormap manipulation, viewpoint manipulation, sequencing | |
| ''//Persson et al. (2006, ~GeoVis lit.)//'' | | |representation, algorithms for the creation of a representation, database, arranging simultaneous views, dynamic linking, temporal dimension, three-dimensional, system interaction |
| ''//Peuquet (1994, ~GeoVis lit.)//'' | | |location, time, object|
| ''//Shepherd (1995, ~GeoVis lit.)//'' | |observer motion, object rotation, dynamic comparison, dynamic re-expression, brushing | |
| ''<<cite Shneiderman1996>>'' | |overview, zoom, filter, details-on-demand, relate, history, extract |one-dimensional, two- dimensional, three- dimensional, temporal, multi-dimensional, tree, network |
| ''<<cite Ward2004>>'' | |navigation, selection, distortion |screen, data, data structure, attribute, object, visualization sturcture |
| ''<<cite Wehrend1990>>'' |identify, locate, distinguish, categorize, cluster, rank, compare, associate, correlate | |scalar, nominal, direction, shape, position, spatially extended region or object, structure|
| ''<<cite Yi2007>>'' |select, explore, reconfigure, encode, abstract/elaborate, filter, connect | | |
| ''<<cite Zhou1998>>'' |associate, background, categorize, cluster, compare, correlate, distinguish, emphasize, generalize, identify, locate, rank, reveal, switch, encode | | |
In Roth's discussion, he reviews several concordances and discordances across the extant taxonomies:
*//identify// and //compare// are the most common objectives
*most objective taxonomies have higher-level categorizations (e.g. <<cite Zhou1998>>, while some are not (e.g. <<cite Yi2007>>)
*objective and operator taxonomies are often hard to delineate
*//brushing// is the most common operator, often semantically related with //selection// and one of: //highlight, shadow highlight, delete// (?), or //label//. Also related to //linking//.
*//focusing// is defined in many different ways: providing more detail, being synonymous with //filtering//, or a second step of //brushing//
*there is ambiguity related to changing/altering the encoding/symbolization: Roth sorts these into "altering the map information that is symbolized", "altering the type of cartographic representation that is displayed", and "altering the graphic parameters of the cartographic representation".
*//viewpoint// operators are related to //distortion, navigation, observer motion, object rotation, panning/re-centre, re-projecting, viewpoint manipulation, zooming//
*many operand taxonomies vary between being //type-centric// and //state-centric//, while others vary between data operands and representation operands
!!!!Comments & Questions
*obvious bent toward ~GeoVis
*high-level and low-level taxonomy cross-cuts the objective category; operator category is low-level
*Note that the ''objective'' category is at the stage of ''forming an intention'' and not at ''forming the goal''. Roth describes intentions as being "formalized at an increased level of precision from the broader goals". He also states that users may not have intentions for all goals, such as for the broader goal of ''exploration''.
*Roth claims that extant taxonomies (or interaction?) align only with these 3 stages of the the seven stages of action. What about taxonomies of goals / higher-level tasks, such as <<cite Amar2004>>? <<cite Dibiase1990>> describes higher level tasks.
!![Roth2012a] - earlier draft or [Roth2012]
<<cite Roth2012a>> asks about interaction in geographical/cartographic visualization: what is interaction? why is it needed? who needs it? when/where is it needed? how should it be provided? These six questions are addressed in turn. The //why// question treads on familiar ground, referring to the generation of ''insight'' via interaction, <<cite Tukey1977>>'s definitions of EDA (and <<cite DiBiase1990>>'s ~GeoVis extensions to it)., and surveying high-level abstract tasks in geovisualization:
*revealing unknown insights to presenting known insights
*visual thinking to visual communication
*revealing anomalies, patterns, trends that were previously unknown, leading to new insight: new hypotheses, ideas, explanations, conclusions
*knowledge-based insight and spontaneous insight (<<cite Chang2009>>).
*''sensemaking'': collection, exploration, evaluation, presentation of evidence for the purpose of decision making
!!!!Comments & Questions
*an earlier draft <<cite Roth2012a>> was submitted to the Journal of Spatial Information Science (JOSIS) ([[available here|http://josis.org/index.php/josis/article/viewArticle/105]]), work is licensed under a Creative Commons Attribution 3.0 License
**the body text is a fairly high-level hodgepodge of theory from HCI, VA, Visualization, and cartography, covering familiar ground - through providing some references to ~GeoVis extensions to Tukey's EDA work that I was previously unfamiliar with; <<cite DiBiase1990>> unavailable online / via UBC library
!![Roth2012b] - ~GeoVis empirically-derived taxonomy of cartographic interaction primitives
[>img(50%, )[<<cite Roth2012b>>'s empirically-derived taxonomy of cartographic interaction primitives|https://dl.dropbox.com/u/6397998/wiki/images/roth_taxonomy.png]]
<<cite Roth2012b>>'s blended theoretical-empirical taxonomy of cartographic interaction primitives from the conceptual framework of <<cite Roth2012>>. The resultant taxonomy was derived from card-sorting exercises on //objective// and //operator// primitives. Roth claims a higher ecological validity than other extant taxonomies:
>//"One limitation of extant taxonomies contributing to their lack of general adoption is that the majority of these taxonomies are not empirically derived, instead relying on secondary sources or personal experience"//
There were 15 cartographic interface designer/developer participants. Cards came from open-ended interviews elicited examples of operator and objective primitives. These were complemented with definitions of these primitives from the literature. Card-sorting results were subject to cluster/similarity analysis.
The results of the analysis suggested a group of ''meta-objectives'', a hierarchy of objective primitives, or ''goals''. The remaining 5 objectives could be sorted by an increasing level of sophistication. The author notes a high degree of variability in responses across the objective primitives. There appeared to be some confusing stemming from different operands (states, data/representation types/abstractions).
Variability was lower for the operator primitives. A common distinguishing trait of operators was between //work interactions// and //enabling interactions//.
In summary, the taxonomy has 4 dimensions: goals, objectives, operators, and operands. More refinement and empirical evaluation is planned, along with card sorting of the results of this study. It is hoped that the taxonomy could be used to prescribe operators in the context of objectives and operands.
*''goals''
**''procure'': //retrieve// information
**''predict'': //forecast// based on current conditions
**''prescribe'': //decide// based on current conditions
*''objectives''
**''identify'': //examine// and //understand// a single feature
**''compare'': //determine// similarities and differences between 2 or more features
**''rank'': //determine// order or relative position of 2 or more features
**''associate'': //determine// the relationship b/w 2 or more features
**''delineate'': //organize// features into a logical structure
*''operators''
**''work''
***''re-express'': //set/change// representation/form
***''arrange'': //manipulate// the layout of (multiple) representation(s)
***''sequence'': //order// the layout of representations
***''re-symbolize'': //set/change// design parameters w/o changing representation/form
***''overlay'': //adjust// the features included
***''re-project'': //set/change// projection of representation
***''pan'': //change// the centre of the representation
***''zoom'': //change// the scale/resolution of the representation
***''filter'': //alter// the representation such that features shown match user-defined conditions
***''search'': //alter// the representation to //indicate// a region of interest
***''retrieve'': //request// details about a feature/region of interest
***''calculate'': //derive// new information about a features/region of interest
**''enabling''
***''import'': //load// a dataset/existing representation
***''export'': //extract// representation, underlying data, system status
***''save'': //store// the representation, underlying data, system status
***''edit'': //manipulate// the underlying data, altering the representation
***''annotate'': add markings/text to //externalize insight//
*''operands''
**''search target''
***''space-alone'': interact with spatial dimensions alone
***''space-in-time'': interact with temporal dimension
***''attribute-in-space'': interact with attribute to see spatial variability
**''search level''
***''elementary'': interact w/ single feature
***''general'': interact w/ several-to-all features
!!!!Comments & Questions
*Presentation of taxonomy: Roth RE. [[The science and practice of cartographic interaction|http://www.slideshare.net/reroth/the-science-and-practice-of-cartographic-interaction-13336159]]. In: GeoInformatics 2012. Hong Kong: June 15th.
*italicized verbs above added for emphasis
*empirically derived interaction primitives show a lot of overlap in our taxonomy, particularly //work operator// primitives; italicized verbs show a higher level structure (//set/change, alter, manipulate//) that are highly interrelated; similarly with objectives, there is //examine, determine//, and //organize//, wherein //compare, rank, associate// could be lumped together
*still the linking between operators, operands, and objectives is not made
!![Sacha2014] - knowledge generation model (VAST 2014)
<<cite Sacha2014>>'s //knowledge generation model// or KGM, specifies an ''exploration loop'' between ''action'', ''computer'' (''data'', ''visualization'', ''model''), and ''finding'', which in turn is part of a larger ''verification loop'' between ''hypothesis'' and ''insight'', which is in turn part of the ''knowledge generation loop'' which results in ''knowledge''. The KGM encompasses the KDD process, the ~InfoVis pipeline, Norman's 7 stages of action. Some high-level characterization of Jigsaw, Tableau, Harvest, and Knime using the KGM are offered as validation.
*''exploration loop'': encompasses "interaction to generate new visualizations or models and analyze the data"
**''action'': a task? an interaction? a goal? both //why// and //how// in <<cite Brehmer2013>>?
**''computer'': (no description)
***''data'': describes facts in "structured, semi-, or unstructred manner"
***''model'': from simple descriptive statistics to complex KDD algorithms
***''visualization'': uses data and models to show relationships in the data
**''finding'': (noun, not verb) an observation without domain context
*''verification loop'':
**''hypothesis'': often vague (e.g. presence and effect of hidden factors), but playing a central role (really vague)
**''insight'': an observation + domain context (not a bad def, in line with <<cite Chang2009>>)
*''knowledge generation loop'': the result of verifying and formulating hypotheses
**''knowledge'': something about "justified belief", trustworthiness (vague)
!!!!Comments & Questions
*super high-level: what value does this add over <<cite Pirolli2005>>
*the KGM is introduced in such a way that makes it hard to follow: ''computer'' first, then ''exploration loop'', then ''verification loop'', and then ''knowledge generation loop'' (bottom-up), whereas top-down is easier to read about.
*terminology swamp: //insights//? //exploroation//? //task//? //action//?
*mischaracterizes the <<cite Brehmer2013>> typology as an //interaction taxonomy//
*who is this paper for? abstract says that their KGM is intended for communication between researchers (which I buy, as long as this means VA researchers, and not ~InfoVis or ~CogSci researchers); intro says the KGM could be used for design and evaluation of systems, but this isn't discussed later in any detail; discussion section refers to use of the KGM for pedagogy.
*poorly written: many sentences don't follow each other. many redundant sections, ESL'isms, typos and mispellings of cited authors
*to read: von Landesberger et al 2014, Endert et al 2014, Roberts et al 2014
!![Saracevic1997] - stratified model of information retrieval interaction
<<cite Saracevic1997>>
!!!!Comments & Questions
*cited by <<cite Marchionini2006>>
!![Scaife1996] - External cognition: how do graphical representations work?
<<cite Scaife1996>>
!!!!Comments & Questions
*
!![Scholtz2004] - Overviewbserving intelligence analysts
<<cite Scholtz2004>>
!!!!Comments & Questions
*
!![Scholtz2006] - Beyond usability: Evaluation aspects of visual analytic environments
<<cite Scholtz2006>>
!!!!Comments & Questions
*
!![Scholtz2009a] - Visual analytics technology transition progress
<<cite Scholtz2009a>>
!!!!Comments & Questions
*
!![Sedig2012] - Towards a characterization of interactivity in visual analytics
<<cite Sedig2012>>
!!!!Comments & Questions
*
!![Sedlmair2012] - design study methodology
<<cite Sedlmair2012c>>
!!!!Comments & Questions
*
!![Sedlmair2013] - DRITW
<<cite Sedlmair2012a>>
!!!!Comments & Questions
*
!![Sedlmair2014] - tasks for parameter space analysis (~InfoVis'14)
<<cite Sedlmair2014>>'s conceptual framework re: parameter space analysis include charcaterization of several tasks, coarsely related to model building, model valiation, and model usage (focus is on latter two):
*''optimization'': find the best parameter combination fiven some objective
*'partitioning': how many different types of model behaviours are there?
*''fitting'': where in the input parameter space would actual measured data occur?
*''outliers'': what outputs are special?
*''uncertainty'': how reliable is the output?
*''sensitivity'': what ranges/variations of outputs to expect with changes of input?
!!!!Comments & Questions
*tasks are more akin to 'targets' or 'what' in <<cite Munzer2014>>
!![Shneiderman2002 - inventing discovery tools
<<cite Shneiderman2002>>
!!!!Comments & Questions
*
!![Shneiderman1996] - Visual Information Seeking Mantra: Overview first, Zoom and Filter, Details on Demand
<<cite Shneiderman1996>>'s task by data type taxonomy describes tasks at two levels: tasks specific to certain data types, and higher-level tasks that incorporate data-type-specific tasks, tasks that generalize to many data types. The mantra //overview-first, zoom and filter, then details-on-demand// incorporates 4 of the 7 tasks; the latter 3 are ''relate'', ''history'', and ''extract''. Many examples (tools, techniques) are given for visualizing and interacting with the various data types and for executing the 7 higher-level tasks.
The 7 higher-level tasks:
*''overview'': a moveable field of view, control contents of the detail view, allow zoom, focus+context; panning and scrolling
*''zoom'': adjusting the field-of-view, zooming to a particular location in a display; control of zoom factor and zoom focus
*''filter'': dynamic querying, control contents of the display, eliminate unwanted items, update with wanted items.
*''details-on-demand'': browse details of a sub-group of items or individual items; a pop-up window with attribute values is a good example of this.
*''relate'': view relationships between items: select an attribute and see items that are similar according to that attribute; select an item and see others that are similar to it; relationship specification - rely on perceptual / Gestalt rules (containment, connection, color coding, highlighting)
*''history'': support for undo/redo, replay, progressive refinement
*''extract'': refers to exporting/saving/printing sub-collections of the original/derived data, query parameters, settings; together with ''history'' supports ''analytical provenance'' and monitoring one's own progress
Tasks by data type:
*''1D'': basic tasks: counting (find number of items), filtering (see items having certain attributes), details-on-demand (provide detail in focus area with less information in surrounding context area)
*''2D'': subsumes 1D tasks; find adjacent items (relate), containment of one item by another (relate), paths between items (compare?/relate)
*''3D'': subsumes 1D and 2D tasks; adjacency plus above/below and inside/outside relationships, understanding position and orientation when viewing items, resolving occlusion issues
*''temporal'': subsumes 1D tasks plus: determine start/end time, find periods of overlap (co-occurrence), find events occurring before, after, during
*''multi-dimensional'': finding patterns of variables, gaps, outliers, resolving disorientation, occlusion
*''tree'': subsumes basic 1D tasks applied to items and links; determine how many levels in the tree, how many children does an item have, examine types of objects at different tree depths, examine breadth/depth
*''network'': subsumes tree tasks; examine shortest/less costly paths, network traversal
!!!!Comments & Questions
*No explicit linking between data-type tasks and 7 higher-level tasks - that exercise is left to the reader
*No related work in task taxonomies given
*''overview'' has a dependency on ''zoom'' (overview+detail is 2 levels)
*alludes to even higher-level tasks / user goals: ''reveal global properties'', ''exploration'', ''finding'', ''sorting'', ''presenting relevant items''
*temporal tasks omit finding cyclical / repeating patterns
*is ''compare'' a task? the question of adjacency is interesting because it could be posed as either a ''relate'' question (what is adjacent to A?) or a ''compare'' question (are A and B adjacent / how far are A and B apart?)
*Aside from basic tasks (''overview, zoom, filter, details on demand''), many data-type tasks are just ''relate'' tasks. None of the data-type tasks refer to ''history'' or ''extract''
*in <<cite Roth2012>> meta-taxonomy
!![Shrinivasan/vanWijk2008/2009] - Exploration Awareness / Analytical Reasoning
<<cite Shrinivasan2008>>'s CHI paper (and later in <<cite Shrinivasan2009>>) is about visualization history tracking and knowledge externalization, however it bases the design and evaluation of a prototype application for performing these activities on existing models of exploration / analytical reasoning. It calls upon the sensemaking/information foraging work of <<cite Pirolli2005>> and <<cite Card1999>>, refers to opportunistic information gathering or ''berrypicking''. The process is unsystematic, evolving, emergent, and aims to construct, contradict, or confirm claims. This involves ''seeking patterns, outliers'' via interaction. It also refers to ~Johnson-Laird and Byrne's stages of reasoning theory: ''model construction'', ''revision'', ''falsification'', wherein the latter stages alternative models are sought to refute/refine the original model. Based on this survey, the authors posit that the high-level stages of visualization use must involve:
*externalizing evidence, hypotheses, assertions, causal links
*organizing evidence to support/refute a claim
*review and revise the exploration process
*link externalized evidence to support claims
*present findings
!![Silva2007] - Provenance for visualizations
<<cite Silva2007>>
!!!!Comments & Questions
*
!!!!Comments & Questions
*Section on analytical reasoning contains pointers to RW; no novel analytical reasoning / high-level taxonomy is provided
*is the exploration process (3rd bullett) cycles of the first two bullets (not made explicit)?
!![Smith2002] - Taxnomies / typologies
<<cite Smith2002>> on taxonomy vs. typology:
>//For present purposes, the key characteristic of a typology is that its dimensions represent concepts rather than empirical cases. The dimensions are based on the notion of an ideal type, a mental construct that deliberately accentuates certain characteristics and not necessarily something that is found in empirical reality (Weber, 1949). As such, typologies create useful heuristics and provide a systematic basis for comparison. Their central drawbacks are categories that are neither exhaustive nor mutually exclusive, are often based on arbitrary or ad hoc criteria, are descriptive rather than explanatory or predictive, and are frequently subject to the problem of reification (Bailey, 1994).//
>//A second approach to classification is taxonomy. Taxonomies differ from typologies in that they classify items on the basis of empirically observable and measurable characteristics (Bailey, 1994, p. 6). Although associated more with the biological than the social sciences (Sokal Q Sneath, 1964), taxonomic methods essentially a family of methods generically referred to as cluster analysis-are usefully employed in numerous disciplines that face the need for classification schemes (Lorr, 1983; Mezzich Q Solomon, 1980).//
!!!!Comments & Questions
*my sense is that in both papers (Multi-Level Tasks and DRITW), we are classifying abstract tasks, and abstract tasks are concepts rather than empirically observable domain tasks. Given this, Smith (and Bailey?) would say we that we've developed a "typology", a "useful heuristic and systematic basis for comparison".
!![Spence2007] - ~InfoVis and interaction
<<cite Spence2007>>: ''Interaction modes'' (continuous, stepped, passive, and composite interaction). ''Interaction spaces'' (continuous, discrete).
The task is one of 'whittling down' and gaining insight (p.16): a subtask of data mining and decision support.
On intention, one could be ''learning'' (''exploring''), ''seeking'' (''finding''), interacting in an ''opportunistic'' or ''involuntary'' manner. ''Browsing'' (''perusal'') can occur in any of these situations, and refers to the perception and interpretation of content, including navigational cues.
The interaction framework of Spence's ch.5 is based in <<cite Norman1988>>'s ''stages of action'' / Action Cycle. Much of what follows covers low-level interaction: ''navigation'', ''sensitivity'', making ''dynamic queries'' and evaluating residue or information scent. He also discerns between interactive and static displays, discrete and composite interaction techniques.
Interaction can be defined by a "palette of techniques and concepts": Norman's Action Cycle, visual dynamics, affordances, interaction modes, navigation, sensitivity, residue, and scent.
!!!!Comments & Questions
*in <<cite Yi2007>> RW
*Not so much about tasks other than ''browsing'' and the 4 intention types. More about low-level interaction.
!![Springmeyer1992] - Characterizing Scientific Discovery
<<cite Springmeyer1992>>'s taxonomy of the scientific data analysis process, though grounded in the domain of scientific visualization, can apply equally as well to information visualization. It is superior to <<cite Winckler2004>>'s taxonomy in that it doesn't attempt to define tasks at the interface or sub-task level. It goes beyond interacting with visualization tools and discusses all aspects and phases of the scientific data analysis process at the abstract level, while bearing in mind opportunities for interactive systems to fill in gaps at these other phases. It recognizes that visualization is merely a means, not an ends to the process, and usually precedes hypothesis verification and further mathematical inquiry. Images are a by-product
They conducted several interviews and observation sessions ([[Interaction Analysis]], [[Contextual Inquiry]]), with 10 scientists working in a 7 domains: physics, biochemistry, numerical analysis. They aimed to presuppose as little as possible, and observe regardless an activity was supported by computer tools. Categories emerged from these sessions (observations being richer than interviews), becoming chronological lists, and structure arose our of these categories:
''Scientific data analysis'': distilling large amounts of measured/calculated data into simple rules/parameters, characterizing phenomena under study
*''Investigation''
**''Interacting with representations'': generating, orientation (cycling, translating, transforming), queries (high/low specificity), comparison, classification: further control for data exploration, particularly for quantitative information
**''Applying math'': calculations (estimations, slope approximations, transformations), deriving new conditions, generation of statistics (i.e. ANOVA)
**''Maneuvering'' (organizing the data): navigation (mechanics of getting data into a system), data management / culling: can be distracting
*''Integration of Insight''
**''Maneuvering'' (see above)
**''Expression of ideas'': recording, describing, (decision making): make judgements, conjectures, deductions: communication
Finally, the authors have several design recommendations:
*how can tools support scientists in integrating insight into their base of knowledge and experience?
*integrate calculations directly
*facilitate active exploration (qual. and quant.)
*capture the context of analysis
*link materials from different stages of a study
*minimize unnecessary or distracting navigation requirements
*support for culling large data sets: flexible, not distracting
*need for domain-specific knowledge
!!!!Comments & Questions
*Superior to <<cite Winckler2004>>'s taxonomy: doesn't adhere to a rigid grammar of tasks and subtasks, user tasks and application tasks.
*future work: collaborative scientific data analysis, very large simulation data
*scientific data analysis process is often not linear, so a chronological ordering of these tasks is doubtful
*methodology sounds awfully close to [[Grounded Evaluation]]
!![Stephenson1967] - The Play Theory of Mass Communication
>//"Mass communication allows people to become absorbed in subjective play"//
!!!ch. 4: play theory
<<cite Stephenson1967>>'s theory posits that all forms of mass media serve the purpose of mutual socialization, to give people "something to talk about", and that maximizing social interaction in the process of digesting mass media can be enjoyed for its own sake. Selective mass media consumption is associated the concepts of "communication-pleasure" and "convergent selectivity", while work-related media consumption is associated with "communication-pain" and "social control". Communication-pleasure holds that play has little gain for the player except in self-enchantment.
As examples, Stephenson considers why we consume documentary films, or attain enjoyment from sensational news stories of violence and scandal. Why does this behaviour appear in both socialized (read: educated) and "feebly-socialized" (read: non-educated) news readers?
News-reading may be explained by the reader's belief in objects of functional pleasure (related to Freud's reality principle, as opposed to his pleasure principle), that despite the lack of an immediate gain from newsreading, the potential for delayed serendipitous and social use of the knowledge attained brings the news reader pleasure.
>//"The reading is undertaken without any expectation of anything, of any rewarrd of any kind, other than being able to say later, if asked, that it was enjoyable; and if studied,one might indeed find the news-reader's self is all the better for it newsreading is a communication-pleasure, sans reward brings no material gain and serves no "work" functions, but it does induce cerain elements of self-enchantment"//
Play certainly means many different things in different cultures, and is conflated with many terms in English. It is not always seen as a direct opposite to "work". It is thereby useful to distinguish types of play, according to Caillois:
*''agon'': agonistic play involving 2 sides (e.g. football)
*''alea'': play involving games of chance (e.g. lotteries)
*''mimicry'': acting, pretending
*''ilinx'': producing dizziness (e.g. swings, carousels)
the ways of playing:
*''paideia'': "primitive, pure play of carefree gaiety, uncontrolled fantasy, and the like" - play theory typically falls under this category
*''ludus'': formal play, e.g. games with rules, conventions; involves patience, development of skill
*''wan'': quietly sensual Chinese way of playing (e.g. polishing jade)
!!!ch. 11: ludenic theory of newsreading
>//"one has to concentrate to read a book but not to read a newspaper is unreliable, like gossip; yet, like gossip, it encourages a sense of belonging to a community. Even so, it is the antithesis of literature and of spirituality. Nor does a newspaper deal very seriously with serious problems; the important news of today is old and forgotten tomorrow."//
Newsreading is by some scholars considered to be governed by Freud's pleasure principle, one of self-actualization. He observes that people read most avidly what they already know about - and this needs to be explained, since rationally, there would be no need to spend one's leisure time reading a newspaper if you already knew what it contained.
''Apperception'' is the "readiness to perceive this or that in relation to prior systems of interest", a preselection, assumes the indiviudal is "a complex of interests, all active and vibrant, with feelers out all the time under appropriate conditions, ready to recieve instantly whatever ties in with a prior interest."
Reading a newspaper is voluntary, a temporary interlude, the "reader is in a sense disinterested in what he is reading", it casts a "spell of deep absorption". Sophisticated newsreading, contemplative rather than scatterbrained, takes on the character of a formal game, reading it in sequence or in a particular regular pattern, where the newsreader creates his own order, "commanding his own grasp of things in the world". One often first picks up a paper, scans the pictures and headlines, returning later to read articles of interest seriously. The initial part of this sequence is "pure play".
>//"Behind the pages there are moments when we seize upon things in an authentice makign of self become committed to this or that in ludenic newsreading. This is without thought of gain and serves no function but to give enjoyment; but there is still the possibility that, at moemnts, in true leisure, one finds something for onself. Reading is well recognized as a place for this self-projection."//
!!!ch. 15: play theory broadly considered
*play is distinguishable from work, it is disinterestd, has attributes of pretense, is self-sufficient, a voluntary interlude.
*play is directed by convergent selectivity, which concerns fads, manners, fashions, taste, and the like; explains responses to advertising, drama, art; mass media offers opportunities for convergent selectivity.
*communication-pleasure, to be discerned with communication-pain, is enjoyment, contentment, serenity, delight; "in attendance upon it is a certain enhancement of self-existence".
!!!!Comments & Questions
*cited by IR scholars: <<cite Toms2000>>, <<cite Case2008>>
*see also: Stephenson, W. //Ludenic theory of newsreading//. Journalism Quarterly, XLI(1964),367-374.
*''apperception'' in self-interested play analogous to ''serendipity'' in discovery and scholarly inquiry?
*swap out "newsreading" above with "using visualizations"
*//"television can sell soap but not, it seems, citizenship"// - G.D. Wiebe (1958-59): "The ~Army-McCarthy hearings and the public conscience", public opinion quarterly IV (pp 409-502)
!![Tang2003a] - taxonomy of tasks and visualizations for casual interaction of multimedia histories
<<cite Tang2003a>>
!!!!Comments & Questions
*
!![Terveen1995] - human computer collaboration
<<cite Terveen1995>>
!!!!Comments & Questions
*
!![Thomas/Cook2005] - Illuminating the Path - ch.2: The Science of Analytical Reasoning
In the second chapter of PNNL's //Illuminating the Path// research agenda, <<cite Thomas2005>> break down analytical reasoning, encompassing information foraging and [[Sense-making]]. It also encompasses a broader definition of the iterative and collaborative analytical process in general, including an overview of the types of analytical tasks (assessments, forecasts, development of opinions), and artifacts involved. [[Reasoning Artifacts]], [[Analytical Discourse]], and [[Sense-making]] are discussed with regards to their role in the analysis process, what is the state of the art, and what technology needs are recommended. The low-level roles of perception and cognition and the analytical processes that depend on them are also discussed. The chapter's final section pertains to collaborative analytic processes, reviewing work from the CSCW field.
The analytical process as defined here is similar/conforms to <<cite Pirolli2005>>'s cycles: determining how to address the issue being posed, what resources to use, planning to allocate time and resources to meet deadlines, gathering information and incorporating prior knowledge ([[Reasoning Artifacts]], create abstractions based on these artifacts, generating multiple explanations (alternate hypotheses) based on convergent and divergent thinking, forms judgments based on evidence gathered, checks assumptions and biases, and considers alternatives. Finally, the analyst presents or reports their judgments, recommends action.
Visual analytics and information visualization amplifies cognition, playing an important role in [[Analytical Discourse]] and [[Sense-making]], serving several functions:
*''increasing resources'': high-bandwidth displays, allowing parallel perception (unlike text), graphical rather than symbolic processing (offloading cognition to perception), expanding working memory, expanding information storage
*''reducing search time'': locality, high density, spatially-indexed
*''enhancing pattern recognition'': recognition instead of recall, abstraction and aggregation, visual schemata for organization, enhance patterns at all levels: values, relationships, trends
*''allowing perceptual inferences'': obvious problems / entities are apparent due to many pre-attentive perceptual processing, graphical computations possible
*''support interactivity''
The authors specify the needs of future technology to support analysis, as well as the needs of the research and development process that leads to this technology. Visually-based methods must be developed for the entire analytic process, from the gathering of artifacts to the weighing of alternative hypotheses. The need for a wider range of evaluation techniques is also among these recommendations: task analysis, field studies must complement lab studies and back-to-back testing. Low-level basic research must also be conducted to better understand the interaction between perception, cognition, attention, and visual analytics tools. In parallel to the above, call for the research and development of mixed initiative systems for assisting the human user in their analysis. Further development of collaborative tools is also called for.
!!!!Comments & Questions
*Draws from many fields of research to create an outline for future R&D (Cognitive Science, Business Intelligence, Perceptual Psychology, CSCW).
*Once you read a single subsection of this book, you realize there's a clear formula to each section: here's what's known with regards to subject x / here's what's been done (or what's the state of the art), and what are the recommendations for improving this process or better-integrating the process into further R&D.
*Definition heavy, many terms used to describe the analysis process and what it encompasses.
*There's an obvious lean towards applications in the intelligence and homeland security sector, as this is their mandate. Therefore it places emphasis on uncertainty and decision-making under time constraints and with various risks. Exploratory analysis and/or creative problem solving without strict consideration for time costs and risks is not a focus of the PNNL's project.
**This is increasingly obvious when discussing constraints of strict time pressures to come up with decisions under uncertainty, and the mention of deception in the data
*Taken together, descriptions of characteristics of how ~InfoVis amplifies cognition can be used as design/evaluation heuristics, albeit at a level without sensitivity to high-level or domain tasks
!![Thomas/Cook2006] - A visual analytics agenda (related TCGA article to //Illuminating the Path//)
<<cite Thomas2006>>
!!!!Comments & Questions
*
!![Toms1999] - serendipitous information retrieval (LIBR)
<<cite Toms1999>> describes motivation for open-ended exploration:
>"//significant evidence exists to support the concept that people also acquire information that was never sought and about which the individual may have had no predisposition."//
Serendipitous information retrieval comes about via affordances and by virtue of having prior knowledge (sagacity). However, interpreting this behaviour can be difficult:
>//"when the interaction was not guided by an objective, user decisions seemed less definitive and less predictable"//
Citation from <<cite Case2008>>: Toms, EG (1998). "What motivates the browser?" In Exploring the Contexts of Information Behaviour. In proc. //Intl. Conf. Research in Information Needs, Seeking and Use in Different Contexts//, pp. 191-208:
>//"There was no need, no anomalous state of knowledge and no knowledge gap evident. This was simply an information gathering experience without expectations or predicted outcome novelty stimulated curiosity (and thus exploration)."//
On the means of acquiring information:
>//1. from the search for information about a well-defined and known object(s);//
>//2. from the search for information about an object that cannot be fully described, but will be recognized on sight; and,//
>//3. from the accidental, incidental, or serendipitous discovery of an object.//
!!!!Comments & Questions
*Toms cited in <<cite Case2008>> in the context of play theory for describing information behaviour
!![Toms2000] - understanding browsing of text
<<cite Toms2000>> reports on an experiment in which two groups of participants read related news articles, where one group was given a learning objective and another was not; the two groups' interactions and usage of the articles varied.
The terms ''browsing, navigating, exploring, way-finding, travelling, orienteering, foraging, grazing, wandering, surfing'', and ''skimming'' are often conflated. Browsing is sometimes described both as a means and as an ends, as either the goal or the method by which the goal is achieved. One definition states that browsing is "an activity in which one gathers information while scanning an information space without an explicit objective". Browsing is not searching, and yet both lie somewhere on a continuum of information retrieval/seeking behaviours. Browsing relies on the personal knowledge and prior experience (a "prepared mind") of the one performing the browsing, along with the affordances of the data, its topography (text, in Toms' article).
Browsing relies on juxtapositions of content in time and space, presented within a context that stimulates the user. The intent of Toms' study was to determine what aspects of an interface are important or useful for browsing, whether these be virtual landmarks or menu structures, recommendations or suggested content. The former supports the way-finding or navigation flavour of browsing while the latter supports exploration and creativity. Toms compared several interface features in a multi-session mixed-factor study of online news reading, in which users were given an implicit browsing task or an explicit search task (a control condition), counterbalanced across subjects and across the 4 sessions.
Those with no goal (implicit browse task) examined more content and more of the information space, however their navigation/selection criteria was not easily predicted (though hierarchical menus afforded this). They found that structured menu navigation was helpful for determining one's location / path through the system, but diversions were how the browsing process was primed. These diversions need be salient and should encourage meandering by which serendipitous discovery could come about. Many claimed to have found articles of interest that they may have not known about or thought to look for. This was deemed by participants as "successful browsing". This outcome is associated with finding something useful not anticipated at the outset, involving accidental discovery and sagacity to recognize something of value.
>//"An unanswered question from this current research is whether browsing can truly be defined as a "task" and measured using the same types of measures, such as recall, precision and fallout, that are applicable to typical search tasks. Browsing in fact may not be a quantifiable, defined task whose success is determined solely by its resolution. Browsing may be more like news reading, more in line with the ''play theory of <<cite Stephenson1967>>''. The success of browsing may be in the experience itself and not in the outcome."// (emphasis added)
!!!!Comments & Questions
*Summarized in <<cite Toms1999>> (see above).
*Relevance of Stephenson's ''play theory'' (<<cite Stephenson1967>>) in visualization? How does Leckie et al's work- or task-driven model of information behaviour (see <<cite Case2008>>) explain serendipitous discovery? It seems to explain casual information consumption.
*I'm generalizing browsing of text to browsing of data. How much of this behaviour transfers?
*Some parallels with scientific discovery ("chance favours the prepared mind"), see <<cite Klahr1999>>, <<cite Kuhn1962>>
!![Tory/M欬er2004] - rethinking vis: high-level taxonomy
<<cite Tory2004>> outlines the relationships between tasks with different visualization design models. Their classification of tasks breaks down into whether ''spatialization'' is constrained (~SciVis vs. ~InfoVis) and whether the design model (conceptualization of a system) is assumed to be ''continuous or discrete'' (with or without structure):
| !Design Model | !Spatialization |>|
|~| ''Given'' (~SciVis) | ''Chosen'' (~InfoVis) |
| ''Continuous'' | find spatial relationships, spatial regions of interest | find numeric trends |
| ''Discrete'' | ''values'': find patterns (clusters, outliers) |>|
|~| retrieve item details, filter/exclude items |>|
|~| ''structure'': analyze connectivity relationships |>|
!!!!Comments & Questions
*Table adapted from Fig. 2, overlap b/w cells omitted (continuous tasks overlap with discrete value tasks, finding numeric trends can also be performed when spatialization is constrained, finding spatial relationships/regions of interest can also occur with discrete models)
*~SciVis and ~InfoVis represented by this taxonomy
*task classification bears most similarity to <<cite Shneiderman1996>>'s task-by-data type taxonomy
*taxonomy in paper title refers to taxonomy by an algorithm //design model// (rather than by data) - which they claim to conceptually match to human tasks to a greater degree than data-type taxonomies, as the user interacts with the data via the design model, which are assumptions about the data and how algorithms operate on the data.
!![Trafton2000] - Cognitive Task Analysis
<<cite Trafton2000>> conducted a [[Protocol Analysis]] at different stages of expert meteorologists' workflow, according to a previously conducted [[Cognitive Task Analysis]], corresponding to different stages of producing a weather forecast.
''Method'': the [[Observational Study]] depicted, including observations by domain experts, a recording of utterances using the [[Think Aloud Protocol]], the use of video recording, and a protocol analysis tool (//~MacSHAPA//), allowed researchers to code the interactions along several dimensions:
*visualization type in use: picture, chart, graph, text
*utterances / written statements reflecting visualization usage: goal statement, information extracted or inference made, brief-writing
**secondary categorizations: qualitative vs. quantitative, integrated vs. non-integrated (multiple visualization sources), source (visualization or QMM: qualitative mental model)
''Results'': the researchers documented the meteorologists' protocol based on their [[Protocol Analysis]]. The expert users first attained a qualitative big-picture view of the data with no explicit detailed data extraction (qualitative or quantitative). The researchers observed how qualitative mental models were constructed, verified, and adjusted based on the integration of several complex visualizations, and how quantitative information was extracted from their qualitative mental models at the brief-writing stage, which relied heavily on domain expertise to convert qualitative knowledge into quantitative information. They proposed that expert users of complex visualizations use heuristics to deal with the large amount of data they encounter: data extraction was integrated across several sources, it was predominantly qualitative and directed by a goal, rather than quantitative and serendipitous.
They speculate that the qualitative mental model is imagerial and spatial, subject to alignment, metric, and rotation errors. They believe their findings generalize to many domains in which expert users must make a prediction based on an integration of many data sources.
!!!!Comments & Questions
*An example of a qualitative lab study with realistic data, representative users, and an observational method
*task analysis drove the protocol coding (i.e. not open coding, but research-driven coding)
**The [[Cognitive Task Analysis]] and its results weren't discussed at length - the task stages and the evidence for them was agreed upon by representative users and domain experts, but how it was conducted and how its results were analyzed were not discussed.
*Section 4.3 - practical implications: read: design implications - a shorter section than anticipated considering the publishing venue - suggesting that systems support the comparison and integration between different visualizations, and that an intelligent agent could retrieve secondary sources of information while a related visualization is being used for extraction.
*Given the date, this is likely one of the first qualitative observational studies conducted for analyzing the information retrieval and integration process. It corresponds with <<cite Lam2011>>'s first scenario of trying to understand the user's workflow (regardless of the presence of a visualization tool or tools).
!![Trafton/Trickett2001] - A new model of graph and visualization usage
<<cite Trafton2001>>
!!!!Comments & Questions
*
!![Treinish1999] - function-based data model for visualization
<<cite Treinish1999>>
!!!!Comments & Questions
*
!![Tukey1977/1980] - EDA
<<cite Tukey1977>> and <<cite Tukey1980>> are on the subject of exploratory data analysis; the latter is a short 3-page //American Statistician// article that refers to the former, his book entitled //Exploratory Data Analysis//.
''EDA'' is more than descriptive statistics, it requires flexibility and an attitude, a //"willingness to look for what can be seen"//. Confirmatory data analysis ''CDA'', can be automated, but depends upon EDA.
!!!!Comments & Questions
*Extended by <<cite Dibiase1990>> (~GeoVis literature) to include a ''synthesis'' stage between ''confirmation'' and ''presentation''.
*<<cite Tukey1980>> has a snarky, pedagogical tone - relating to the teaching of EDA and CDA (and ordering thereof)
!![Tufte1997] - Visual Explanations
<<cite Tufte1997>>:
>//"We thrive in information worlds because of our marvelous and everyday capacity to ''select'', ''edit'', single out, structure, ''highlight'', ''group'', pair, ''merge'', harmonize, ''synthesize'', ''focus'', ''organize'', condense, reduce, boil down, choose, ''categorize'', catalog, ''classify'', list, ''abstract'', scan, look into, idealize, ''isolate'', ''discriminate'', ''distinguish'', ''screen'', pigeonhole, pick over, ''sort'', integrate, blend, ''inspect'', ''filter'', lump, skip, smooth, chunk, average, approximate, ''cluster'', ''aggregate'', ''outline'', ''summarize'', itemize, review, dip into, flip through, ''browse'', glance into, leaf through, skim, refine, ''enumerate'', glean, synopsize, winnow the wheat from the chaff and separate the sheep from the goats."//(p.50)
!!!!Comments & Questions
*emphasis added to tasks already included in our survey
*quoted in <<cite Liu2010>>, in "beyond interaction techniques" - some are active and some are passive in many visualization systems (<<cite Spence2007>>)
!![Tweedie1996] - Externalising abstract mathematical models
<<cite Tweedie1996>>
!!!!Comments & Questions
*
!![Tweedie1997] - Characterizing interactive externalizations
<<cite Tweedie1997>>: Interaction types (manual, mechanized, instructable, steerable, and automatic) and directness (direct and indirect manipulation)
>//Questions can be asked of this data at three different levels i.e. about: a single item, a set of items, or the whole set. These questions can relate to objects or attributes.//
Distinguishes static and dynamic comparisons.
>//It is important to explicitly represent Input and Output relations. This provides an externalization of the current state of the interaction. This is important if the user is to engage in a dialogue with the visualization.//
Outlines the DIVA system for describing data-centric interactions with visualizations.
!!!!Comments & Questions
*in <<cite Yi2007>> RW
!![Upson1989] - application visualization system
<<cite Upson1989>>
!!!!Comments & Questions
*
!![Valiati2006] - High- and ~Low-Level Task Taxonomies for Eval.
<<cite Valiati2006>>'s BELIV '06 provides a taxonomy of high- and low-level tasks to be used in evaluation. The taxonomy borrows heavily from other related work, which they survey in detail.
Their taxonomy integrates high- and low-level tasks, as well as analytic, cognitive, and operational tasks A hierarchy cannot be established between them (nor a workflow?):
*''identify'' - clusters, correlations, categories, properties, patterns, characteristics, thresholds, similarities, differences, dependencies, independencies, uncertainties, variations
*''determine'' - mean, median, variance, standard deviation, amplitude, percentile, sum, proportions, differences, correlation coefficients, probabilities, other statistics
*''visualize'' - //n// dimensions, //n// items, data, domain parameters / attribute information / metadata
*''compare'' - dimensions, items, data, values, clusters, properties, proportions, positions / locations, distances, graphical primitives
*''infer'' - hypotheses, rules, trends, probabilities, cause / effect
*''configure'' - normalization, classification, filtering, zoom, dimensions order, derived attributes, graphical primitives
*''locate'' - items, data, values, clusters, properties, position / locations, distances, graphical primitives
''Methodology'':They conducted a small evaluation on 2 visualization tools constructed using the ~InfoVis toolkit (//Parallel Coordinates//, //~RadViz//). A small number of study participants recruited from the computer science department (and one domain expert, a biologist) were recruited. A set of four directed questions were used to assess the validity of the taxonomy. They use the benchmark cars data set.
!!!!Comments & Questions
*A long related work, given the total article length, a good job relating to previous taxonomies (the novelty of this task taxonomy isn't very apparent - is it because it doesn't distinguish between high- and low-level tasks, as well as analytic, cognitive, and operational tasks, between orders of tasks, hierarchies, dependencies?)
*Their taxonomy is the main contribution, preceded by their decent lit review. Their own study is not compelling enough, were the problem sets chosen knowing that they would easily validate their taxonomy? What is the breadth of these problems? Also, the benchmark cars data was used - how generalizable are these findings with novice users and a simple toy data set and only 2 forms of visualizations? What was the training procedure?
**I appreciated how their didn't specify dependencies or progressions of tasks, or lengths of tasks, suggesting that tasks can very in length and when they occur in a sequence
!![Van Ham/Perer2009] - Graph task taxonomy
<<cite Ham2009>>
!!!!Comments & Questions
*
!![Vicente1999]
<<cite Vicente1999>>
!!!!Comments & Questions
*cited by R3 (primary):
>//Though the authors did not refer to the following book and other relevant literature, I strongly suggest them to refer to the work domain analysis in the following book (<<cite Vicente1999>>. The authors will enjoy it.//
!![Viegas2004] - Digital artifacts for remembering and storytelling
<<cite Viegas2004>>
!!!!Comments & Questions
*
!![Ward/Yang2004] - Interaction spaces
<<cite Ward2004>>: ''operator-based taxonomy'': navigation, selection, distortion
*state-centric objective-by-operand taxonomy (<<cite Roth2012>> categorization)
*interaction operators (navigation, selection, distortion), interaction spaces (screen-space, data value-spaces, data structure-space, attribute-space, object-space, and visualization structure-space), and interaction parameters (focus, extents, transformation, and blender)
!!!!Comments & Questions
*in <<cite Roth2012>> meta-taxonomy
*in <<cite Yi2007>> RW
!![Ware2013] - Ware on Tasks in Information Visualization
<<cite Ware2013>>'s text on Information Visualization does not address tasks in great detail. There are two chapters dedicated to low-level and high-level tasks, respectively.
In the //Interacting with Visualizations// chapter, there are three loops of activity that define interaction with a visualization:
*''data manipulation'' loop: described low-level HCI, concrete, selecting, positioning, moving objects, relying on hand-eye coordination, hover queries, vigilance tasks
*''exploration and navigation'' loop: establishing a cognitive spatial model of a navigation space, viewpoint control and locomotion (wayfinding and navigation); overview+detail, focus+context, and scale (panning+zooming) - recalls <<cite Shneiderman1996>>'s mantra; rapid interaction techniques (linking, brushing)
*''problem solving'' loop (not addressed in the interaction chapter): high-level, abstract, hypothesis forming and refinement
In the //Thinking with Visualizations// chapter, the third loop is tangentially addressed, referencing <<cite Pirolli2005>>'s sensemaking model. However, it mainly deals with memory models and knowledge costs, limits to visual working memory, and eye movements. There is a section on visual problem solving, which breaks the process down into the following hierarchy:
*''problem solving'' strategy: set up problem solving steps, requirements, and problem context (mainly non visual); redefine if necessary
**''visual query'' construction: translate problem to one that could be expressed as a visual subproblems, simple patterns or objects that can be held in visual working memory (path finding, cluster identification, correlation identification, trend estimation, outlier detection and characterization, magnitude estimation, quantity estimation, target detection, structural pattern / distribution characterization, etc.)
***''pattern-finding'' loop: establish a visual search pattern according to target visual query
****''eye movement'' control loop: conscious eye movement to areas of interest defined by query pattern
*****''intrasaccadic image-scanning'' loop: unconscious super-low-level detection of proto-objects
Each part of this hierarchy is addressed in detail, with implications for design and interaction costs described.
He also has several sections on conceptual understandings of complex visualizations, such as those resulting from concept maps, multidimensional scaling, and trajectory mapping.
The final section of the chapter addresses creative problem solving and creative thinking (high-level tasks), which previous work (//Cognition//, 3rd ed. by Matlin, M. W. (1994)) characterized as involving stages of //preparation, production, and judgment//:
*''preparation'': acquire background information on problem, ''exploratory data analysis'', pattern discovery, loosely defined visual queries
*''production'': generate possible solution set, rejecting or refining tentative hypotheses, quantity over quality, support for "sketchiness" and tentative interactions, allow for multiple interpretations, rapid iteration and discarding of ideas
*''judgement'': analysis of potential solutions, quality control, hypothesis rejection, many judgments not visual (e.g. formal hypothesis testing)
!!3rd edition ch. 11 (visual thinking algorithms):
The ten visual thinking algorithms as explicitly stated in the 3rd ed. (implicit in the 2nd ed) are:
*visual queries
*pathfinding on a map or diagram
*reasoning with a hybrid of a visual display and mental imagery
*design sketching
*brushing
*small pattern comparisons in a large information space
*degree-of-relevance highlighting
*generalized fisheye views
*multidimensional dynamic queries with scatter plot
*visual monitoring strategies
!!!!Comments & Questions
*2 chapters, one on interaction with visualization (low-level), another related to thinking with visualization (high-level)
*Also an appendix section on task specification/identification for user studies: make sure it's theoretically interesting and commonly used in real applications, or at least representative of those used in visualization interfaces; he states that tasks should have "a simple and clear user response" (i.e. a yes/no response); this limits what tasks can be evaluated in a user study (interaction usability, graphical perception tasks)
*problem-solving strategy step is addressed briefly, however this is where tasks are defined
*is ''EDA'' defined as pattern discovery and loosely-defined visual queries?
!!!!3rd ed. Comments & Questions
*more visual thinking algorithms since publication (20ish)
*what is overlap of visual thinking algorithms and tasks? Are VTAs propertieds of tasks, tools, or their combination? Totally orthogonal?
!![Weaver2007a] - Patterns of coordination in Improvise
<<cite Weaver2007a>>
!!!!Comments & Questions
*TM 11.21.12 discussion
!![Wehrend/Lewis1990] - ~Problem-Oriented Classification
<<cite Wehrend1990>> aim to classify visualization problems as a lookup problem of (data) object classes and operations (user tasks). Problems break down into subproblems. The latter part of problems (operations) doesn't get much treatment in the article, and are given as a flat list only. Very similar list to <<cite Roth1990>> (in brackets):
*''identify'' (lookup value)
*''locate'', ''distinguish'', ''categorize'', ''cluster'' (determine)
*''distribution''
*''rank'', ''compare'' (within and between relations)
*''associate'', ''correlate''
!!!!Comments & Questions
*in <<cite Roth1990>> but not <<cite Wehrend1990>>: ''index/sort'' a structure by an element
*Low-level tasks without much description; more in Wehrend's thesis (90 problems gathered from the literature)
*in <<cite Roth2012>> meta-taxonomy
!![Wilkinson 2005] - The Grammar of Graphics
<<cite Wilkinson2005>> (ch. 17 - control) discerns ''building'' graphics from ''exploring'' interactive graphics, the latter comprising of ''filtering'' (categorical/continuous/multiple/fast filtering), ''navigating'' (zooming/panning/lens), ''manipulating'' (node dragging/categorical reordering), ''brushing'' and ''linking'' (brush shapes/brush logic/fast brushing), ''animating'' (frame animation), ''rotating'', ''transforming'' (specification/assembly/display/tap/2 taps/3 taps)
On not being a taxonomy (p.14)
>//Classification for its own sake, however, is as unproductive in design as it is in science. In design, objects are only as useful as the system they support. And the test of a design is its ability to handle scenarios that include surprises, exceptions, and strategic reversals. classifications] may be useful for developing interfaces but contributes nothing to a deeper understanding of graphs. Customary usage and standards can blind us to the diversity of the graphics domain; a formal system can liberate us from conventional restrictions.//
On the grammar itself (ch. 2), he described the grammar as a means of tranfsorming data to graphic, likening it to a recipe. It is object-oriented, comprised of sets, relations, functions, graphs, compositions, transformations, algebras, variables, varsets, frames. It describes how to create variables, apply algebra, apply scales, compute statistics, construct geometries, apply coordinates, compute aesthetics. The notation itself is presented.
!!!!Comments & Questions
*in <<cite Yi2007>> RW
!![Wilson1999] - models of information behaviour - LIBR
<<cite Wilson1999>>, a model of information seeking: see <<cite Case2008>>
!!!!Comments & Questions
*Represented in <<cite Case2008>> survey
!![Wilson2000] - human information behaviour - LIBR
<<cite Wilson2000>>
!!!!Comments & Questions
*
!![Winckler2004] - ~Bottom-Up Task Taxonomy
<<cite Winckler2004>> generated a hierarchical taxonomy that encompassed user goals, abstract tasks, interaction tasks, and application tasks (rendering tasks) supported by the system. Based on this taxonomy, they are able to generate scenarios for completing user goals by enumerating the paths of lower-level tasks, traversing the branches of the taxonomy. Once these scenarios are enumerated, they can be used as heuristics to evaluate the usability of a system, or to compare functionality and usability between several systems. The authors argue that scenarios generated from their taxonomy used for evaluation are more appropriate than ad-hoc or informal evaluation using ungrounded scenarios or ill-defined heuristics, which may be biased. They believe their method of creating a taxonomy (identify user goals, interaction mechanisms available, & rendering functions supported, then relate all three) will account for all possible user goal scenarios, and that these scenarios will be rationalized.
Abstract tasks were adapted from earlier taxonomy by <<cite Wehrend1990>>. They use the <<cite Zhou1998>> taxonomy of tasks in their CTT (Concur Task Tree notation for evaluation, <<cite Paterno2002>>), distinguishing between task types/actors and levels:
*''abstract task'': locate / identify / distinguish / reveal / cluster / emphasize / explore (an odd mix of <<cite Zhou1998>>'s presentation intents and visual tasks)
**can be found at the goal level (locate), abstract level (identify, explore), interaction level (search, identify), and visual presentation level (emphasize, reveal cluster)
*''user task'': ways of achieving visual tasks in <<cite Zhou1998>>: identify by name, portray, individualize, profile
**occurs at the interaction level
*''application task'': focus / isolate / reinforce / expose / itemize / specify / separate / outline / individualize, highlight, colour, zoom, (ways of achieving visual tasks, by the application rather than the user)
**occurs at the visual presentation level
*''interactive task'': select / finish (exit)
**occurs at the abstract or interaction level
Their case study involves comparing 4 scenarios of a "locate system file" goal between a hyperbolic tree browser and a treemap browser.
!!!!Comments & Questions
*Step 1: producing unambiguous user goals and logical decomposition of subtasks seems trivial in the paper, the difficulty of this in reality is not addressed.
**What about scenarios in which the user has no explicit goal other than to explore a dataset, or the goal is ill-defined, such as "attempting to gain insight"
*Checking all possible scenarios of a task can run upwards of thousands of scenarios - the authors case study only checks 4 scenarios. How do they advise one should proceed given a goal with several thousand possible scenarios for completing the goal?
*distinctions between tasks, particularly abstract and user, and between application and interaction, or user and interaction, aren't as clear-cut as I expect the authors imagine. A user may engage in an abstract user task without interaction (adjusting a mental model, perhaps)
*The authors do not address the scalability of this method for large, complex systems
*paper/typos:
**CTT acronym not defined until p. 3 (not helpful)
**Table/Figure numbering off.
!![Wiss1999] - An empirical study of task support in 3D information visualizations
<<cite Wiss1999>>
!!!!Comments & Questions
*
!![Yang2014] - Understand users' comprehension and preferences for composing information visualizations (TOCHI)
<<cite Yang2014>>
!!!!Comments & Questions
*
!![Yi2007] - Interaction in ~InfoVis
<<cite Yi2007>> describes a list of interaction tasks used in InfoVis systems and techniques, toward the purpose of developing a taxonomy of interaction techniques to support analytic reasoning (<<cite Thomas2005>>). They discuss what "interaction" means in the context of Visualization, implying manipulation and change occurring in a visual representation; representation in ~InfoVis is often difficult to separate from interaction. They compare existing taxonomies, which are either to low-level (system-centric) or high-level (user-goal). They believe an interaction taxonomy to be the bridge between these levels. They mention that no existing taxonomy provides each descriptive, generative, and evaluative power (<<cite Beaudouin-Lafon2004>>).
They reviewed existing taxonomies, read technique papers, and used a variety of techniques to generate their list of interactions. They collected 311 interaction techniques. They first used affinity diagramming but this broke down, as some interactions did not group easily. Instead, they grouped interactions by user goal/intent (how was this known?). They formed 7 categories, which are likely cross-cutting and not exhaustive.
*''select'': mark items, track before/after, identify locations of interest, add placemark
*''explore'': examine subset, move to other area; panning, ~Direct-Walk
*''reconfigure'': change spatial arrangement, reveal hidden characteristics, jitter, change arrangement/alignment, provide different perspective, compare, reduce occlusion, make arrangement more suitable for mental model, Dust+Magnet
*''encode'': alter fundamental representation (''Visualize'' in <<cite Heer2012a>>?); change visual appearance, change representation: colour, shape, size, orientation
*''abstract/elaborate'': adjust level of abstraction; overview+detail, details-on-demand, drill-down, tool-tips, zooming (semantic, geometric)
*''filter'': change set of items to be shown, specify range conditions, dynamic queries controls (alphasliders, rangesliders, toggle buttons), showing nearby/similar items; greying-out filtering items but not removing
*''connect'': highlight, relationships, associations, show hidden/similar items and relationships; linking and brushing; cross-cutting with ''explore'' category
*''other'': undo/redo, change configuration / layout / settings (''organize'' in <<cite Heer2012a>>).
They expect that some are at a higher-level abstraction than others: ''compare'' is not included, but is associated with ''filter'', ''reconfigure'', and ''encode''.
!!!!Comments & Questions
*How do you determine user intent? Is this their task?
*Argue that these are still low-mid level, above system-specific interactions, but it's still a gap to move from these interactions to "task".
*Not a taxonomy, but a list of 7 interaction types.
*what are "have multiple perspectives" or "gain insight" - are these mid-level tasks? consequences of analysis? high-level abstract goals?
*a full taxonomy would include interface-specific interactions and higher-level user goals, but these interaction categories are still a far leap from these goals; another intermediary layer is in between.
*in <<cite Roth2012>> meta-taxonomy
!![Yi2008] - Characterizing Insight
<<cite Yi2008>>'s BELIV '08 workshop paper attempts to answer questions relating to how and when insight is gained, rather than adding to the pile of previous work which attempted to explain what //insight// is and how it is characterized - which remains poorly understood (<<cite Mayr2010>> refers to this as a "//black box//"). It is a meta-review, not of evaluation methods or of qualitative research, but of gathered instances in previous work where insight was gained and via what means. It therefore fits into the notion of sensemaking and information foraging as insight is considered to be the central element of <<cite Pirolli2005>>'s sensemaking and information foraging model: //information > scheme > ''insight'' > product//. They posit that insight is not the product (as in <<cite Pirolli2005>>'s model), but a midpoint for more cycling and iterations back and forward towards a product (sensemaking is often retrospective).
They surveyed quotes from 4 books, 2 chapters, 34 papers, and determined that insight is gained during several overlapping but distinct processes: when providing an overview (big picture), when adjusting the level of abstraction (i.e. grouping / filtering), when detecting patterns, and when the data matches one's mental model. They also accounted for barriers to gaining insight: usability problems, poor motivation, lack of background/domain knowledge or training
The authors plan to develop insight-based heuristics for evaluation and design, as identifying how and when insight is gained is not sufficient for these tasks. They also plan to investigate how insight is gained with information visualization tools compared to other means of gaining insight.
!!!!Comments & Questions
*An //insight// on //insight// paper? (a few funny sentences such as this appear throughout)
*A "When and how? - not What?" meta-review similar to <<cite Lam2011>> re: evaluation scenarios - "when and how to choose an evaluation method", not what methods can I choose from?
**"When and how does insight happen?", rather than "what is insight?"
*Their section on barriers to insight does support the notion that evaluation should be carried out with real users in real situations with real data that they feel motivated about.
!![Yuen2009] - human computation
<<cite Yuen2009>>
!!!!Comments & Questions
*
!![Zhang1993] - interaction b/w perceptual & cognitive processes in a distributed problem solving task
<<cite Zhang1993>>
!!!!Comments & Questions
*
!![Zhang/Norman1994] - Representations in distributed cognitive tasks
<<cite Zhang1994>>
!!!!Comments & Questions
*
!![Zhang1996] - representational analysis of relational information displays
<<cite Zhang1996>>
!!!!Comments & Questions
*
!![Zhou/Feiner1998] - Automated Visual Discourse
<<cite Zhou1998>>'s CHI paper contains a number of abstractions. The central abstraction is that of a ''visual task''. A taxonomy of visual tasks can be defined based on higher-level ''presentation intents'': what the user wants to achieve via visual discourse (interaction) with the tool. Visual tasks interface ''low-level visual techniques'' and presentation intents along two dimensions: ''visual accomplishments'' and ''visual implications''. The latter refers to visual organization (grouping, attention, composition), signalling (encoding), and transformation that lead to accomplishments: the communication/integration of visual information.
Their taxonomy can generalize between interfaces used for information seeking and those used for communication. The list of visual acts and "ways of achieving them" (low-level interface interaction?) are:
*''Associate'': collocate, connect, unite, attach
*''Background'': background
*''Categorize'': mark distribution
*''Cluster'': outline, individualize
*''Compare'': differentiate, intersect
*''Correlate'': plot, mark compose
*''Distinguish'': mark distribution, isolate
*''Emphasize'': focus, isolate, reinforce
*''Generalize'': merge
*''Identify'': name, portray, individualize, profile
*''Locate'': position, situate, pinpoint, outline
*''Rank'': time
*''Reveal'': expose, itemize, specify, separate
*''Switch'': switch
*''Encode'': label, symbolize (quantify, iconify), portray, tabulate, plot, structure, trace, map
Arranged in a taxonomy by presentation intents (visual accomplishments):
*''inform'':
**''elaborate'': emphasize, reveal
**''summarize'': associate, background, cluster, compare, correlate, distinguish, generalize, identify, locate, rank
*''enable'':
**''explore'':
***''search'': categorize, cluster, compare, correlate, distinguish, emphasize, identify, locate, rank, reveal
***''verify'': categorize, compare, correlate, distinguish, identify, locate, rank, reveal
**''compute'':
***''sum'': correlate, locate, rank
***''compute'': correlate, locate, rank
Examples and descriptions of visual accomplishments and implications follow, using the list of visual tasks.
!!!!Comments & Questions
*Individual presentation intents and visual tasks are not described in detail
*Several layers of abstraction at a level below user goals but above perceptual tasks makes for a confusing read. (they refer to ''highlight, zoom, include'' as low-level visual techniques as opposed to abstract visual tasks).
>//The visual task taxonomy covers high-level visual organization and low-level visual symbol encoding tasks.//
*Perception > visual task > visual encoding/technique (way of achieving the visual task)?
*Is a ''visual task'' low-level or a blend of low-mid level tasks? It seems that presentation intents and user goals are more of what we're looking to define.
*Visual accomplishments refer to visual tasks for achieving presentation intents (fig 2), while visual implications refer to a //"particular type of visual action that a visual task may carry out"// (eek!)
*claim that mapping between visual tasks and presentation intents are domain-independent: but can they help us to evaluate? Do presentation intents map clearly to user goals? Presentation intents / visual accomplishments still seem too low-level.
*Visual implications section on visual organization, signalling, and transformation goes on a long tangent into Gestalt psychology, with examples and using sequences of visual tasks to explain them
*Not specific to ~InfoVis but to all visual discourse / interactions, with an emphasis on presentation and communication.
*in <<cite Roth2012>> meta-taxonomy
!![Ziemkiewicz2011] - locus of control influences compatibility with visualization style
<<cite Ziemkiewicz2011a>>
!!!!Comments & Questions
*
!![Ziemkiewicz2012] - Within/Between Graph Workflows / Individual Differences
<<cite Ziemkiewicz2012a>>'s CHI note describes a short observational study of 4 immunobiologists using a scatterplot-based visualization tool called GenePattern. Their recorded observations of users' interactions were coded with a combined scheme of <<cite Springmeyer1992>>'s scientific data analysis and <<cite Amar2005>>'s low level visual analytical tasks. A second coding past was quantitative, counting instances and durations of visualizations used. Their findings revealed two markedly different interaction strategies, despite similar user goals and context. A //within-graphs// interaction strategy focused on individual data points rather than layouts, emphasizing information in context and visual connections between parts of the data, manual linked brushing and highlighting, with visualization as the primary output of analysis. A //between-graphs// interaction strategy was one in which switching between graphs was frequent, with each graph treated as a step in the analysis, constantly refreshing the view to see the data from different perspectives. The different strategies may indicate subtle differences in user goals, not evident at the start of the observation session: using visualization to increase confidence in results, to ensure the validity of the data, and efficiency (viewing lots of data).
!!!!Comments & Questions
*+qualitative coding strategies for observation sessions from <<cite Springmeyer1992>> and <<cite Amar2005>> a good use of prior theory; but could this be constricting?
*-small N (4); users evidently had different goals that were not communicated before the session began, despite being at a similar stage of analysis / workflow, and working with similar data.
*visualizations should record users' analytical processes, an analytical provenance, a history of past views or manual bookmarks such as in <<cite Heer2007a>>'s sense.us system.
*-analysis strategies may change within a session, within a workflow, reflecting local or global changes in user goals
*-cites <<cite Wickham2010>>'s graphical inference method, yet in the context of sequential analysis of valid data from different perspectives, not a parallel analysis of a valid view amongst random views. Not sure what was intended by this citation, perhaps a misreading/over-generalization of the graphical inference method.
*cites <<cite Pirolli2005>>, aims to bridge the sensemaking process to the immunobiology domain.
*Unsure where the CHI-level contribution is in this work - reads like a WIP / pilot study, not a lessons-learned paper
!References
<<bibliography Bibliography-TaskTaxonomy showAll>>
!!Speaker: Colin Ware (UNH Data Visualization research lab)
*viusal thinkign design patterns (design patterns' origins in architecture)
*epistemic actions: eye movements, mouse clicks, etc. - simple actions: actions executed to seek knowledge - brushingm dynamic queries, zooming - the essence of interactive visualization
*transforming problem tasks / cognitive tasks into visual queries (breaking down a problem into a visual pattern search)
*visual working memory - implications from change blindness, we don't see the world in rich detail
*VTDP example 1: humpback whale data
**analysis tool eventually used for presenting / making policy decisions, not for analysis and pattern discovery
**trackplot: pattern-finding time requirements decreased by > 100x
*VTDP example 2: social network degree of relevance highlighting - determining what the users' tasks were based on eye moevements and movements
**visual query > epistemic action > highlighting of short paths > visual pattern query > if item of interest found, another epistemic drilldown action (epistemic actions once per second)
*VTDP example 3: searching for small patterns in large spaces (islands of information) - zoomable user interfaces; artificial task - comparisons via zooming vs. comparisons via extra windows; after 3 or more items, it's quicker to use multiple zoom windows than zooming, hige differences in errors
*the design process:
**cognitive task analysis and data affordances > design with Aperture 仠recommenders: interaction (VTDPs) + data mappings prototypes for evaluation + cognitive walkthroughs and anlayst feedback > product
**⫠w/ oculus (Toronto company)
*agile visualization: design for cognitive efficiency:
**design is closer to "Overview First, Zoom + Filter, Details-on-Demand" mantra (Shneiderman) than ACT-R (Anderson)
!!Notes
*so far identified 20~ or so VTDPs - what's the mapping from tasks to VTDPs? what granularity of problem or cognitive tasks?
A qualitative, implementation-stage qualitative evaluation technique. The user is asked to describe what he/she is doing while using a tool/system, what is believed to be happening, what is being attempted. This occurs as a researcher observes (<<cite Dix2004 bibliography:Bibliography>>)
''Pros'': Simple. It can be conducted in lab or field settings, providing both high-level and low-level information. Requires little or no equipment (i.e. pencil and paper notes).
''Cons'': Hard to analyze. It is intrusive, highly-subjective and selective, time-consuming, medium-level expertise with the domain/context, or tools. Observation changes the behaviour, therefore biasing results. Dependent on recording method.
!!Sources
<<bibliography>>
|~ViewToolbar|closeTiddler closeOthers +editTiddler > fields syncing permalink references jump externalize easyEdit|
|~EditToolbar|+saveTiddler -cancelTiddler deleteTiddler|
''Speaker'': Sheelagh Carpendale (Canada Research Chair, Information Visualization)
Tuesday, January 15 (11:30 am - 12:30 pm)
Room 4127, Earth Systems Building (ESB), 2207 Main Mall (actually at SFU via [[WestGrid|http://westgrid.ca/about_westgrid]])
!!Notes:
3 high-level arguments:
*parallel b/w computing science and visualization: push toward the personal, personalized visalization
*information age: information barons and serfs, a new internet dark ages, we need to rethink this, we need to take ownership over the data, take control
*big data: growing size of data, much of it is social data, about people, like Facebook data;
This talk is in response to shifts in goals in the Vis/VA field:
*enhancing cognitive abilities: representational transformation
*search (targeted / exploratory, serendipitous search)
*efficiency vs. aesthetics: e.g. casual, personal visualization (e.g. bar charts in the bathroom)
*accuracy vs. ease:
*security and privacy: targeted reading of information
*fit with work vs. everyday social life: tools need a sense of belonging
*decision support: e.g. persuasive computing, sustainability
*sharing
Four themes of this talk:
*''methodology'': reflections from a multidisciplinary background vertical (science) vs. horizontal (arts); observations for design: inspiration comes from watching people
**the problem of seeing what we expect to see; granularity different levels of observation; examples:
***tabletop territoriality
***collaborative information analysis - how do you make collaborative VA atemporal?
***gestures in context (aquarium tabletop gestures both on and off the table)
***hand drawn thinking diagrams
***keystrokes for enriching communication, usage violated expectations
***~EMDialog - Emily Carr tree-ring diagram
***[[FatFonts|http://fatfonts.org]] - levels of detail, nested characters
**perception studies (e.g. 3D on tabletops)
*''representation'' and ''presentation'' (on the screen space)
**the undistort lens, algorithmically agnostic, cartographic projections, fisheye views, back-mapping
**~TouchWave (Dominikus Baur)
*''interaction'' (see last year's TVCG paper on Beyond mouse and keyboard (B. Lee))
**models of interaction; temporal model of interaction: intent > action > reaction > feedback
***HCI: WIMP, direct manipulation, instrumental (the first half of this timeline)
***~InfoVIs: tasks, goals, interaction
**edge bundling, interactive link legends
**Vis on whiteboards: what you draw is what you get? Tableau too complex, can it be simpler with sketching?
**alternate search methods for visual exploration of the web (Marian Dୠglobal voices search
**serendipity - library digitalization / Bohemian Bookshelves, multiple visualizations
**eco-balance (Polenti/Baur)
**personal visualization (daily activity timeline), message wave
!!Questions:
*diversity of evaluation methods; with each piece of research, think of evaluation questions relating to the research question
**a tendency toward full-scale evaluation (e.g. video coding)
*toolkit work (forthcoming TEI Feb. paper) on video / graphics processing work
Referring to a number of quantitative evaluation techniques, encompassing laboratory-based [[User Observation Studies|Laboratory Observation]], controlled [[User Studies|Controlled Experiments]], and [[Inspection Methods]]. Appropriate for evaluating human performance and interface usability during mid-to-late design stages and in the post-design stage (summative evaluation). Discouraged at early stages of design, as these methods focus on problems and may inhibit innovation.
Sources:
<<cite Greenberg2008 bibliography:Bibliography>>
!!Serendip: Topic ~Model-Driven Visual Exploration of Text Corpora
Eric Alexander, Joe Kohlmann, Michael Witmore, Robin Valenza, Michael Gleicher (<<cite Alexander2014a bibliography:VIS2014>>
*''CPSC 547''
*still want to read the documents rather than present a model of the documents. A tool as a lens, not as a replacement for reading
*English scholars collab. They approached researchers with idea of topic models
*multiple levels of abstraction: corpus, document, passage, word
*issues of scale: many documents, long documents
*support serendipitous discovery ࠬa Bohemian Bookshelf
*corpus viewer, text viewer, rank viewer
*live demo fail in corpus view. Not ready for prime time?
*topics over the course of the a document in the document viewer. Tracing trends from corpus to passage
*[[vep.cs.wisc.edu/serendip/|http://vep.cs.wisc.edu/serendip/]]
*moving away from topic models in FW
*Q: why numbered topics and not semantic labels? Aggregation of topics?
!References
<<bibliography>>
(''Note'': first ~2 min of feedback not recorded)
@@color:#0000dd;''PI'': vis in general is a lot of stuff is domain-specific, data is domain-specific. extremely important to think of tasks on a higher level, but its extremely hard to get it right. You're doing the right thing, to think about different levels.@@
@@color:#0000dd;''PI'': An Interesting BELIV 2014 paper on the definition of "task" [Rind et al]. An analyst may have a different sense of the word "task" than a vis evaluator. A typical evaluation paper will have a task section that's very low-level (at the level of clicks). A designer may also think about tasks in a different way. This was something not quite reflected, though it's good to see if [this characterization of tasks] applies to different [scenarios] of analysis, design, and evaluation.@@
@@color:#0000dd;''PI'': I would like to see more about how you think of tasks in this typology and how this fits in terms of how other people think of tasks. It's an important first step that you are applying it [in different scenarios], though in the end you want others to apply it. If you were working in an industry where it's often the case where there's a human factors person who does the pre-design study and talks to people. If they apply your [typology] and follow it, write out the tasks as you describe it, and then hand that off to the designers, would that be satisfying for the designers? Would it be enough for them to do something / target their designs? And then later when the designers tell the evaluators "here's the tasks I designed for", and they go to do their user testing, is it actionable into study tasks.@@
''MB'': Yes. How much gets lost in translation between the three groups of people?
@@color:#0000dd;''PI'': So once that all of that is done, then you can definitely state that in your ~PhD.@@
@@color:#dd0000;''CC'': you've said that 22 people have cited this already.@@
''MB'': A few people have used it in analysis-type projects.
@@color:#dd0000;''CC'': From a dissertation point of view you could go back and say "these are the ways in which people have used this already" and analyze their analysis, validate it that way. I know that I've found it useful. Especially the 2x2 characterization of search, using it in a survey paper right now.@@
''MB'': Addendum chapter, a typology revisited (sub)chapter is planned. Bioinformatics journal paper [by Mirel and G沧]. Network task paper from ~EuroVis [Saket et al].
@@color:#dd0000;''CC'': I liked the slide you showed where you were coding interviews of energy analysts. You're the only one doing that coding right? That would be another way to validate [the typology], in that could you achieve inter-rater reliability between coders using the typology. Could you hire an RA to go through the data too and train them to use the typology.@@
''MB'': this coding was done while an intern. Others were familiar with the code set but I was the only one coding it.
@@color:#dd0000;''CC'': Did the interviewees confirm it? Sanity check?@@
''MB'': Yes, though of course the typology is absent of any domain-specific jargon, that's when we go back to translate these tasks into domain language? We initially had 3-4 contender tasks that we thought would be important, though it turned out that one task was carried out much more prevalently than others.
@@color:#0000dd;''PI'': In perspective four [Pulse], to what extent did the model help you come up with the designs, the different types of designs? You glossed over it by saying "here are the tasks" and "here are the screenshots of my designs", but I'm sure there's lots of thinking [that went into them] and variation in the design that came from a higher-level description. That's why I wonder if you gave it to somebody else. Imagine you have the ~InfoVis [grad] course you are running and you say "here are the people we talked to, they have these tasks and these tasks, build something to support them". What will students come up with and would [the task descriptions] be helpful for them, would it be satisfying enough or would they still have unanswered questions? (''MB'': And how much variation would there be in terms of design?).@@
@@color:#0000dd;''PI'': One of the problems is is that if you're the one who did the interviews and came up with the model, you have lots of contextual information that may not be described and yet factor into these designs.@@
@@color:#0000dd;''PI'': That's why I was thinking of interaction. Interaction is the other component, you can support tasks by encoding alone, but interaction is often a part of it. They're both very tricky things to design for, and there's so much variation.@@
''MB'': an interesting idea, since so many vis courses leave it open-ended as to what the project is. The student has to decide what the task is themselves, or maybe they go and use the VAST challenge data, but I think giving someone a task framed like I have here, using words from the typology, we can see how much variation there is and if they accomplish the task in the end.
@@color:#dd0000;''CC'': [Re: How questions from each perspective]: Are you hoping that your dissertation will provide answers to these how questions or are you actually going to do them? For example, "how should I study adoption or appropriation in the wild?" Are you trying to contribute a method for doing that or are you trying to contribute an example of studying adoption and appropriation? On what level are you trying to answer this question?@@
''MB'': This question is a longer-term career question. How do we do it? Likely not something I'll address head-on in the dissertation because it is not as central to the discussion of tasks, though out in the wild you have little say over what the tasks are, so it does factor into it.
''CC'': even the third question [about validating task typologies in evaluation] could be a dissertation question in itself, delving into this question, providing different ways to validate [task characterizations] and contributing the methods for validation, rather than the validation of a specific example. But I understand what you're trying to do, to answer these questions for your particular task typology and then addressing these other questions as longer term career questions.
@@color:#dd0000;''CC'': the work that you do draws quite heavily on grounded theory or ethnography.@@
''MB'': that's how it started. It started as being very open-ended, [bottom-up] based on observations from these projects out in the wild, and [top-down] from the literature: we needed a code set.
@@color:#dd0000;''CC'': you asked a question in your document about how do you study a case in which a deployed visualization tool is used briefly and then abandoned. Is that still something of interest to you? That didn't come up in your presentation today.@@
''MB'': It does tie into this question about adoption, studying use as well as non-use. Analyzing visualization failure stories.
@@color:#dd0000;''CC'': do you have a failure story that you're trying to fit into your dissertation?@@
''MB'': In the journalism project we had a bunch of successful case studies where they used the tool and published stories as a result of using it, but there's a whole other group of people who tried it out and stopped using it and we don't really know why it failed, because there's no incentive for people to report why a tool failed. How do you get people out in the wild to say why your visualization didn't work for them? There's a reward structure for everyone who does get your visualization to work.
@@color:#0000dd;''PI'': You could do a thesis about that. It can start from some other cause, like pulling money from the project, something that has nothing to do with your tool. If they still wrote an article about how they didn't use your tool then you could go and figure out why.@@
@@color:#dd0000;''CC'': If it's truly in the wild and they don't have buy-in to actually talk to you, that would be really hard to do. If they are partners / collaborators and they've agreed to try and use it and they don't, then you have the opportunity to go and talk to them. But if they just don't use it, it's tough.@@
@@color:#0000dd;''PI'': I co-wrote this article with Michael Sedlmair about evaluation in industry contexts, where he was trying to deploy stuff at BMW. Part of the adoption problem was hierarchical, so he had to do an interesting kind of quantitative study to show "hey, my tool is 20% faster", and then the engineers said "that much time?", and that got him the approval to deploy stuff, but I think he still wants to go back and do a long term adoption study.@@
''MB'': It's easier in this [Pulse] project because I've already developed a rapport with prospective users and collaborators that I can go back and ask them why it didn't work. But with the journalism project, the tool itself was developed by an industry partner and deployed, and we don't have the same close connection with people who just pick it up.
@@color:#0000dd;''PI'': for a study like that, with adoption, you actually need partners, ideally there should be an industry partner in there.@@
''MB'': so that's why I'm wondering how I could do this type of work, maybe in industry. Is that something that I should be considering and where should I be considering to work and which companies would value this type of work? It's a branding problem for me. I don't exactly know how to brand myself yet.
@@color:#0000dd;''PI'': Yeah I was wondering if you need to be a consultant because you need access to several different industries in order to be able draw more general conclusions as to why visualizations fail or succeed in the wild. I guess you could learn about adoption when people ask you to consult for developing something new and examining what people had before, find out why it was or wasn't adopted and how it was used.@@
@@color:#dd0000;''CC'': I think the method that you are building upon here could be used in situations where they don't have anything yet but they just have a data analysis problem and you need to go in and say "what are the specific tasks that you want to do with this data". People will tell you what questions they *think* they have but I think, through your method, that you'll find different questions. I like that idea.@@
@@color:#dd0000;''CC'': Your document goes into a bit of a different direction, asking the question about how do you visualize data that is aggregated in terms of items and in terms of time. And you're now asking us in your big picture questions about how to frame your dissertation. I think you've got enough in grounding the design in the task typology and the validation of the task typology through design studies that this sounds like a specific visualization design question comes a little bit out of the blue and a bit off topic for me.@@
''MB'': That's why I didn't cover it today, and the document is already dated, several months old, my thinking is already moved beyond that. Initially in my early versions of this talk I had these specific low-level visualization design questions and visual encoding questions and I thought it would take too long to describe in this context here, I really wanted today to have a bigger picture question about how I could use this going forward.
@@color:#0000dd;''PI'': but generally are you not happy with your current structure? Because it seemed quite good to me in terms of how to structure the thesis. The flow seemed pretty good. Is that what you presented to your committee as well?@@
''MB'': It was in May but it has evolved a little bit since then, but they're generally on board.
@@color:#dd0000;''CC'': So the way this worked out was that the Overview project led to this investigation which led to the task typology which is then verified with the energy study? Is that the timeline? (''MB'': yes). It's not necessarily the way you need to tell the story but is that the timeline of how it went? I think that to me sounds like canonically good dissertation in terms of having these specific projects but also the theoretical framing with the typology. You're asking about validation and whether we buy it and stuff. Honestly, a lot of people get away with a lot less in terms of presenting a framework. "Here's my framework, I just made it up because I needed to string my projects together but they get away with it for a dissertation but it doesn't have an impact, the framework itself doesn't become a paper, but you've already got the framework about the paper, so I feel like I don't have a lot more to say. Does anyone else?@@
''MB'': it is an interesting story, because the 2nd (Overview) and 3rd (DRITW) perspectives predated the typology but I had to put them on hold because I didn't know how to move forward without some qualitative data analysis tool. So chronologically it's a bit of a weird dissertation in that way. The design study project [Pulse] is something that's happened since then, beginning a week after last year's VIS conference.
@@color:#00dd00;''RB'': Typologies are always a challenge to publish but they're really useful to have at the time of engagement. Sometimes, they feel like checklist-type things when you're trying to understand if there's a slot in the framework that doesn't match or if there are questions that I haven't asked.@@
@@color:#00dd00;''RB'': ~Post-PhD, you know, if you want to work, there are places like ours [O㓓㝠that do data visualization, that might require a move to T㓓㓮 We do need to work in situations, and I'm sure there are many companies that have this problem (web design and visualization), where you need to understand the tasks and work with the users and find out what it is that they're actually doing. I would still be suspect of the ability to do a hand-off from the notion of a task decomposition to a designer. Simple example that I could give you there is that we've done some visualizations that would have identical task descriptions but the visualization that would work for the target audience would be very different. If I look at financial professionals who are traders, the screens that they use and how they cram in information into the screens versus a web user who is looking at the same information but is a casual user and how much data is simplified in user interfaces and the expectation is two different things, but they are still doing the same task. So I'd be suspect.@@
''MB'': this is very similar to what happened in this last project in which we have very different users, which is why I developed so many different visualizations.
@@color:#00dd00;''RB'': it's very similar to portfolio analysis, whether its a portfolio of building energy or a financial portfolio.@@
@@color:#dd0000;''CC'': This may be a bit of a mean question, but if your typology worked and you're able to isolate what people need through your analysis, why did you have to make 12 different visualizations? Why weren't you able to target it a bit better? I've never made 12 visualizations for domain experts. Maybe I need to. (''MB'': It's a fair question.) I mean you could answer that all 12 visualizations do different things and they liked all of them, but if there are some that were addressing things that they didn't need, why would you need to make them?@@
''MB'': I think the mentality there is that there's a divergent phase of design followed by a convergent phase of design. They [Pulse] have only taken a small subset of those 12 visualizations and put them into their production code project plan for that reason. We realized after showing these to our prospective users that there are some that are used much more than others. We had both absolute data and relative data, the analysts would do different types of comparisons that did require different visual encoding designs for that, and it turned out that one type of comparison is done more than others.
@@color:#dd0000;''CC'': Were your different designs addressing different tasks?@@
''MB'': They were addressing different data abstractions: absolute vs. relative data. There were visualizations around ranking, which is why I initially started thinking about ranking visualizations. It turned out that ranking was done less prevalently in analysis workflows than absolute comparisons, so the ranking
visualization didn't make it into the production code timeline.
@@color:#dd0000;''CC'': I'm trying to relate the typology and the tasks you identified to the 12 visualization sketches that you made.@@
''MB'': I think that would require a longer discussion.
@@color:#dd0000;''CC'': Yes, though maybe clarifying [this relationship] in the dissertation will be something to consider.@@
''MB'': Yes, and I'm not sure if an ~InfoVis or ~EuroVis paper would have this, in which I spell out each visualization design in terms of the typology. Is that useful? Or is this something more supplemental? Is it only useful to me?
@@color:#dd0000;''CC'': I think that I would find it useful to say "we identified these tasks and from *these* tasks we identified *these* sketches, because of course I know you need to make multiple sketches and multiple prototypes in order to get the right one. I think that's widely accepted, especially within HCI, but right now it seems like you came up with the tasks using the typology and then it was from there to 12, and I'm not sure where the connection is.@@
@@color:#0000dd;''PI'': I think it would be really useful to describe also what, a sketch is one thing, but when did you have to deliver more than that? Some descriptions of interactions or some specific design decisions you made to specifically address the task that the people implementing it definitely cannot change, because that always happens right, they take your design and change something. What else did you have to give to them [Pulse developers]? Did they get your task typology when you gave them the designs and what did they do with it, did they look at it? If you could interview people and see to what extent they used it if you gave it to them.@@
@@color:#dd0000;''CC'': Yeah, if you give these designs to them and you don't tell them what it's for and then you ask them and they say "I use this to lookup and compare information" and use the words from your typology then that would be a form of validation. "I found it very useful and I was able to select" or whatever.@@
''MB'': get them away from using domain jargon?
@@color:#dd0000;''CC'': if they see the visualization sketch that you've designed to support a specific task but you don't tell them that it was designed to support that task, you just say "here's a visualization that I designed to support your work", and then you ask them, in what way would you use it? And they come back and say "I found it useful for discovering榱uot;.@@
''MB'': It's similar to PI's idea of giving a task description to a bunch of students in a course and having other students decipher their designs, almost a double-blind review process in a course project. This would require course infrastructure to carry out. It's a good idea, if I start thinking about that now, perhaps for next fall it could be done.
@@color:#dd0000;''CC'': if you get through the ethics process.@@
''MB'': thank you very much.
{{{
@article{Bach2014,
author = {Bach, B. and Pietriga, E. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {5},
pages = {740--754},
title = {{GraphDiaries: Animated transitions and temporal navigation for dynamic networks}},
volume = {20},
year = {2014}
}
@inproceedings{Accorsi2014,
author = {Accorsi, P. and Ber, F. L.},
pages = {123--132},
title = {{HydroQual: Visual analysis of river water quality}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Alexander2014a,
author = {Alexander, E. and Kohlmann, J. and Valenza, R. and Witmore, M. and Gleicher, M.},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {173--182},
title = {{Serendip: Topic model-driven visual exploration of text corpora}},
year = {2014}
}
@inproceedings{Behrisch,
author = {Behrisch, M. and Korkmaz, F. and Shao, L. and Schreck, T.},
pages = {43--52},
title = {{Feedback-driven interactive exploration of large multidimensional data supported by visual classifier}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Bradel2014,
author = {Bradel, L. and North, C. and House, L. and Leman, S.},
pages = {163--172},
title = {{Multi-model semantic interaction for text analytics}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Crnovrsanin2014,
author = {Crnovrsanin, T. and Muelder, C. and Ma, K.-L.},
pages = {33--42},
title = {{A system for visual analysis of radio signal data}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Dietrich2014,
author = {Dietrich, C. and Koop, D. and Vo, H. T. and Silva, C. T.},
pages = {23--32},
title = {{Baseball4D: A tool for baseball game reconstruction \& visualization}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Gomez2014,
author = {Gomez, S. R. and Guo, H. and Ziemkiewicz, C. and Laidlaw, D. H.},
pages = {63--72},
title = {{An insight- and task-based methodology for evaluating spatiotemporal visual analytics}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Member,
author = {Janetzko, H. and Sacha, D. and Stein, M. and Schreck, T. and Keim, D. A. and Deussen, O.},
pages = {13--22},
title = {{Feature-driven visual analytics of soccer data}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Ko,
author = {Ko, S. and Walton, S. and Chen, M. and Ebert, D.},
pages = {83--92},
title = {{Analyzing high-dimensional multivariate network links with integrated anomaly detection, highlighting and exploration}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Landstorfer2014,
author = {Landstorfer, J. and Stange, J.-E. and D\"{o}rk, M. and Wettach, R.},
pages = {73--82},
title = {{Weaving a carpet from log entries: A network security visualization built with co-creation}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Lenz,
author = {Lenz, O. and Keul, F. and Bremm, S. and Hamacher, K. and von Landesberger, T.},
pages = {93--102},
title = {{Visual analysis of patterns in multiple amino acid mutation graphs}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Li2014,
author = {Li, J. and Zhan, K.},
pages = {133--142},
title = {{Vismate: Interacttive visual analysis of station-based observatiion data a on climate changes}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Liu2014a,
author = {Liu, S. and Wang, X. and Chen, J. and Zhu, J. and Guo, B.},
pages = {183--192},
title = {{TopicPanorama: a full picture of relevant topics}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Lu,
author = {Lu, Y. and Kr\"{u}ger, R. and Thom, D. and Wang, F. and Koch, S. and Ertl, T. and Maciejewski, R.},
pages = {193--202},
title = {{Integrating predictive analytics and social media}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Nunes,
author = {Nunes, M. and Rowland, B. and Schlachter, M. and Matkovic, K. and Laprie, A. and Katja, B.},
pages = {53--62},
title = {{An integrated visual analysis system for fusing MR spectroscopy and multi-modal radiology imaging}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Schmidt,
author = {Schmidt, J. and Preiner, R. and Auzinger, T. and Wimmer, M. and Gr\"{o}ller, M. E. and Bruckner, S.},
pages = {153--162},
title = {{YMCA 㠙our mesh comparison application}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Wang,
author = {Wang, F. and Chen, W. and Wu, F. and Zhao, Y. and Hong, H. and Gu, T. and Wang, L. and Liang, R. and Bao, H.},
pages = {103--112},
title = {{A visual reasoning approach for data-driven transport assessment on urban roads}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Wongsuphasawat2014,
author = {Wongsuphasawat, K. and Lin, J.},
pages = {113--122},
title = {{Using visualizations to monitor changes and harvest insights from a global-scale logging infrastructure at Twitter}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Wu2014a,
author = {Wu, W. and Zheng, Y. and Qu, H.},
pages = {143--152},
title = {{BoundarySeer: Visual analysis of 2D boundary changes}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Zahalka2014,
author = {Zah\'{a}lka, Jan and Worring, Marcel and Member, Senior},
pages = {3--12},
title = {{Towards interactive, intelligent, and integrated multimedia analytics}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@inproceedings{Zhao2014a,
author = {Zhao, Jian and Gou, Liang and Wang, Fei and Zhou, Michelle},
pages = {203--212},
title = {{PEARL: An interactive visual analytic tool for understanding personal emotion style derived from social media}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2014}
}
@article{Alsallakh2014a,
author = {Alsallakh, B. and Hanbury, A. and Hauser, H. and Miksch, S. and Rauber, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1703--1712},
title = {{Visual methods for analyzing probabilistic classification data}},
volume = {20},
year = {2014}
}
@article{Beham2014,
author = {Beham, M. and Herzner, W. and Gr\"{o}ller, M. E. and Kehrer, J.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {1},
pages = {1693--1702},
title = {{Cupid: Cluster-based exploration of geometry generators with parallel coordinates and radial trees}},
volume = {20},
year = {2014}
}
@article{Brown2014,
author = {Brown, E. T. and Ottley, A. and Zhao, H. and Lin, Q. and Souvenir, R. and Endert, A. and Chang, R.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1663--1672},
title = {{Finding Waldo: learning about asers from their interactions}},
volume = {20},
year = {2014}
}
@article{Chen2014,
author = {Chen, H. and Chen, W. and Mei, H. and Liu, Z. and Zhou, K. and Gu, W. and Ma, K.-L.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {1},
pages = {1683--1692},
title = {{Visual abstraction and exploration of multi-class scatterplots}},
volume = {20},
year = {2014}
}
@article{Dang2014a,
author = {Dang, T. N. and Wilkinson, L.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1624--1632},
title = {{Transforming scagnostics to reveal hidden features}},
volume = {20},
year = {2014}
}
@article{Ferracutti2014,
author = {Ferracutti, G. and Gargiulo, F. and Castro, S. M. and Bjerg, E. and Gr\"{o}ller, M. E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1913--1922},
title = {{The spinel explorer - Interactive visual analysis of spinel group minerals}},
volume = {20},
year = {2014}
}
@article{Gotz2014,
author = {Gotz, D. and Stavropoulos, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1783--1792},
title = {{DecisionFlow: Visual analytics for high-dimensional temporal event sequence data}},
volume = {20},
year = {2014}
}
@article{Isaacs2014,
author = {Isaacs, E. and Domico, K. and Ahern, S. and Bart, E. and Singhal, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1793--1802},
title = {{Footprints: A visual search tool that supports discovery and coverage tracking}},
volume = {20},
year = {2014}
}
@article{Klemm2014,
author = {Klemm, P. and Oeltze-Jafra, S. and Lawonn, K. and Hegenscheid, K. and Volzke, H. and Preim, B.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1673--1682},
title = {{Interactive visual analysis of image-centric cohort study data}},
volume = {20},
year = {2014}
}
@article{Ko2014,
author = {Ko, S. and Zhao, J. and Xia, J. and Afzal, S. and Wang, X. and Abram, G. and Elmqvist, Niklas and Member, Senior and Kne, Len and Riper, David Van and Gaither, Kelly and Kennedy, Shaun and Tolone, William and Ribarsky, William and Ebert, David S},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1853--1862},
title = {{VASA: Interactive computational steering of large asynchronous simulation pipelines for societal infrastructure}},
volume = {20},
year = {2014}
}
@article{Koch2014a,
author = {Koch, S. and John, M. and Worner, M. and Muller, A. and Ertl, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1723--1732},
title = {{VarifocalReader 䠉n-depth visual analysis of large text documents}},
volume = {20},
year = {2014}
}
@article{Kothur2014,
author = {Kothur, P. and Sips, M. and Dobslaw, H. and Dransch, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1893--1902},
title = {{Visual analytics for comparison of ocean model output with reference data: Detecting and analyzing geophysical processes using clustering ensembles}},
volume = {20},
year = {2014}
}
@article{Krause2014,
author = {Krause, J. and Perer, A. and Bertini, E.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1614--1623},
title = {{INFUSE: Interactive feature selection for predictive modeling of high dimensional data}},
volume = {20},
year = {2014}
}
@article{Madhavan2014,
author = {Madhavan, K. and Elmqvist, N. and Vorvoreanu, M. and Chen, X. and Wong, Y. and Xian, H. and Dong, Z. and Johri, Aditya},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1823--1832},
title = {{DIA2: Web-based cyberinfrastructure for visual analysis of funding portfolios}},
volume = {20},
year = {2014}
}
@article{Mahyar2014,
author = {Mahyar, N. and Tory, M.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1633--1642},
title = {{Supporting communication and coordination in collaborative sensemaking}},
volume = {20},
year = {2014}
}
@article{Malik2014,
author = {Malik, A. and Maciejewski, R. and Towers, S. and McCullough, S. and Ebert, D. S.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1863--1872},
title = {{Proactive spatiotemporal resource allocation and predictive visual analytics for community policing and law enforcement}},
volume = {20},
year = {2014}
}
@article{Cs2014,
author = {Matkovic, K. and Gracanin, D. and Splechtna, R. and Jelovi, M. and Stehno, B. and Hauser, H. and Purgathofer, W.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1803--1812},
title = {{Visual analytics for complex engineering systems: Hybrid visual steering of simulation ensembles}},
volume = {20},
year = {2014}
}
@article{Muhlbacher2014,
author = {Muhlbacher, T. and Piringer, H. and Gratzl, S. and Sedlmair, M. and Streit, M.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1643--1652},
title = {{Opening the black box: Strategies for increased user involvement in existing algorithm implementations}},
volume = {20},
year = {2014}
}
@article{Partl2014,
author = {Partl, C. and Lex, A. and Streit, M. and Strobelt, H. and Wassermann, A.-M. and Pfister, H. and Schmalstieg, Dieter},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1883--1892},
title = {{ConTour: Data-driven exploration of multi-relational datasets for drug discovery}},
volume = {20},
year = {2014}
}
@article{Poco2014,
author = {Poco, J. and Dasgupta, A. and Wei, Y. and Hargrove, W. and Schwalm, C. R.. and Huntzinger, D. N. and Cook, R. and Bertini, E. and Silva, C. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1923--1932},
title = {{Visual reconciliation of alternative similarity spaces in climate modeling}},
volume = {20},
year = {2014}
}
@article{Sacha2014a,
author = {Sacha, D. and Stoffel, A. and Stoffel, F. and Kwon, B. C. and Ellis, G. and Keim, D. A.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1604--1613},
title = {{Knowledge generation model for visual analytics}},
volume = {20},
year = {2014}
}
@article{Sadransky2014,
author = {Sadransky, B. and Cornel, D. and Perdig, R. A. P. and Horv, Z. and Gr\"{o}ller, M. E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1873--1882},
title = {{Run watchers: Automatic simulation-based decision support in flood management}},
volume = {20},
year = {2014}
}
@article{Shi2014,
author = {Shi, C. and Wu, Y. and Liu, S. and Zhou, H. and Qu, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1733--1742},
title = {{LoyalTracker: Visualizing loyalty dynamics in search engines}},
volume = {20},
year = {2014}
}
@article{Stolper2014a,
author = {Stolper, C. D. and Perer, A. and Gotz, D.},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {1653--1662},
title = {{Progressive visual analytics: User-driven visual exploration of in-progress analytics}},
volume = {20},
year = {2014}
}
@article{Sun2014a,
author = {Sun, G. and Wu, Y. and Liu, S. and Peng, T.-Q. and Zhu, J. J. H. and Liang, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
title = {{EvoRiver: Visual analysis of topic coopetition on social media}},
volume = {20},
year = {2014}
}
@article{Sun2014,
author = {Sun, M. and North, C. and Ramakrishnan, N.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1713--1722},
title = {{A five-level design framework for bicluster visualizations}},
volume = {20},
year = {2014}
}
@article{Wang2014,
author = {Wang, Z. and Ye, T. and Lu, M. and Yuan, X. and Qu, H. and Yuan, J. and Wu, Q.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1813--1822},
title = {{Visual exploration of sparse traffic trajectory data}},
volume = {20},
year = {2014}
}
@article{Wu2014,
author = {Wu, Y. and Liu, S. and Yan, K. and Liu, M. and Wu, F.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1763--1772},
title = {{OpinionFlow: Visual analysis of opinion diffusion on social media}},
volume = {20},
year = {2014}
}
@article{Xie2014,
author = {Xie, C. and Chen, W. and Huang, X. and Hu, Y. and Barlowe, S. and Yang, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1743--1752},
title = {{VAET: A visual analytics approach for e-transactions time-series}},
volume = {20},
year = {2014}
}
@article{Yu2014,
author = {Yu, B. and Doraiswamy, H. and Chen, X. and Miraldi, E. and Arrieta-ortiz, M. L. and Hafemeister, C. and Fellow, T Silva},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
pages = {1903--1912},
title = {{Genotet: An interactive web-based visual exploration framework to support validation of gene regulatory networks}},
volume = {20},
year = {2014}
}
@article{Zeng2014,
author = {Zeng, W. and Fu, C.-W. and Erath, A. and Qu, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {1},
title = {{Visualizing mobility of public transportation s`ystem}},
volume = {20},
year = {2014}
}
@article{Zhang2014,
author = {Zhang, J. and Yanli, E. and Ma, J. and Zhao, Y. and Xu, B. and Sun, L. and Chen, Jinyan and Yuan, Xiaoru},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1843--1852},
title = {{Visual analysis of public utility service problems in a metropolis}},
volume = {20},
year = {2014}
}
@article{Zhao2014,
author = {Zhao, Jian and Cao, Nan and Wen, Zhen and Song, Yale and Lin, Yu-Ru and Collins, Christopher},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. VAST)},
number = {12},
pages = {1773--1782},
title = {{\#FluxFlow: Visual Analysis of Anomalous Information Spreading on Social Media}},
volume = {20},
year = {2014}
}
@article{Al-Awami2014,
author = {Al-Awami, A. K. and Beyer, J. and Strobelt, H. and Kasthuri, N. and Lichtman, J. K.. and Pfister, H. and Hadwiger, Markus},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2369--2378},
title = {{NeuroLines: A subway map metaphor for visualizing nanoscale neuronal connectivity}},
volume = {20},
year = {2014}
}
@article{Armstrong2014a,
author = {Armstrong, Z. and Wattenberg, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2132--2141},
title = {{Visualizing statistical mix effects and simpson's paradox}},
volume = {20},
year = {2014}
}
@article{Bae2014,
author = {Bae, J. and Watson, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1973--1982},
title = {{Reinforcing visual grouping cues to communicate complex informational structure}},
volume = {20},
year = {2014}
}
@article{Borgo2014,
author = {Borgo, R. and Dearden, J. and Jones, M. W.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Order of magnitude markers: An empirical study on large magnitude number detection}},
volume = {20},
year = {2014}
}
@article{Boy2014a,
author = {Boy, J. and Rensink, R. A. and Bertini, E. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{A principled way of assessing visualization literacy}},
volume = {20},
year = {2014}
}
@article{Brehmer2014c,
author = {Brehmer, M. and Ingram, S. and Stray, J. and Munzner, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2271--2280},
title = {{Overview: The design, adoption, and analysis of a visual document mining tool for investigative journalists}},
volume = {20},
year = {2014}
}
@article{Chevalier2014,
author = {Chevalier, F. and Dragicevic, P. and Franconeri, S.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2241--2250},
title = {{The not-so-staggering effect of staggered animated transitions on visual tracking}},
volume = {20},
year = {2014}
}
@article{Correll2014b,
author = {Correll, M. and Gleicher, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2142--2151},
title = {{Error bars considered harmful: Exploring alternate encodings for mean and error}},
volume = {20},
year = {2014}
}
@article{Cui2014a,
author = {Cui, W. and Liu, S. and Wu, Z. and Wei, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2281--2290},
title = {{How hierarchical topics evolve in large text corpora}},
volume = {20},
year = {2014}
}
@article{Demiralp2014,
author = {Demiralp, C. and Bernstein, M. S. and Heer, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1933--1942},
title = {{Learning perceptual kernels for visualization design}},
volume = {20},
year = {2014}
}
@article{Duarte2014a,
author = {Duarte, F. S. L. G. and Sikansi, F. and Fatore, F. M. and Fadel, S. G. and Paulovich, F. V.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2063--2071},
title = {{Nmap: A novel neighborhood preservation space-filling algorithm}},
volume = {20},
year = {2014}
}
@article{Fuchs2014,
author = {Fuchs, J. and Isenberg, P. and Bezerianos, A. and Fischer, F. and Bertini, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2251--2260},
title = {{The influence of contour on similarity perception of star glyphs data lines only ( D ) data lines + contour ( D + C ) contour only ( C )}},
volume = {20},
year = {2014}
}
@article{Goffin2014,
author = {Goffin, P. and Willett, W. and Fekete, J.-D. and Isenberg, P.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Exploring the placement and design of word-scale visualizations}},
volume = {20},
year = {2014}
}
@article{Gramazio2014,
author = {Gramazio, C. C. and Schloss, K. B. and Laidlaw, D. H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1953--1962},
title = {{The relation between visualization size , grouping , and user performance}},
volume = {20},
year = {2014}
}
@article{Gratzl2014a,
author = {Gratzl, S. and Gehlenborg, N. and Lex, A. and Pfister, H. and Streit, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2023--2032},
title = {{Domino: Extracting, comparing, and manipulating subsets across multiple tabular datasets}},
volume = {20},
year = {2014}
}
@article{Guo2014,
author = {Guo, D. and Zhu, X.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2043--2052},
title = {{Origin-destination flow data smoothing and mapping}},
volume = {20},
year = {2014}
}
@article{Harrison2014,
author = {Harrison, L. and Yang, F. and Franconeri, S. and Chang, R.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1943--1952},
title = {{Ranking visualizations of correlation using Weber's law}},
volume = {20},
year = {2014}
}
@article{Huron2014c,
author = {Huron, S. and Jansen, Y. and Carpendale, S.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2102--2111},
title = {{Constructing visual representations: Investigating the use of tangible tokens}},
volume = {20},
year = {2014}
}
@article{Isaacs2014a,
author = {Isaacs, K. E. and Bremer, P.-T. and Jusufi, I. and Gamblin, T. and Bhatele, A. and Schulz, M. and Hamann, B.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2349--2358},
title = {{Combing the communication hairball: Visualizing parallel execution traces using logical time}},
volume = {20},
year = {2014}
}
@article{Jo2014,
author = {Jo, Jaemin and Huh, Jaeseok and Park, Jonghun and Kim, Bohyoung and Seo, Jinwook},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2329--2338},
title = {{LiveGantt: Interactively Visualizing a Large Manufacturing Schedule}},
volume = {20},
year = {2014}
}
@article{Kindlmann2014b,
author = {Kindlmann, G. and Scheidegger, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2181--2190},
title = {{An algebraic process for visualization design}},
volume = {20},
year = {2014}
}
@article{Kondo2014,
author = {Kondo, B. and Collins, C.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2003--2012},
title = {{DimpVis: Exploring time-varying information visualizations by direct manipulation}},
volume = {20},
year = {2014}
}
@article{Lex2014,
author = {Lex, A. and Gehlenborg, N. and Strobelt, H. and Vuillemot, R. and Pfister, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1983--1992},
title = {{UpSet: Visualization of intersecting sets}},
volume = {20},
year = {2014}
}
@article{Liu2014,
author = {Liu, Z. and Heer, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2122--2131},
title = {{The effects of interactive latency on exploratory visual analysis}},
volume = {20},
year = {2014}
}
@article{McKenna2014,
author = {McKenna, S. and Mazur, D. and Agutter, J. and Meyer, M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2191--2200},
title = {{Design activity framework for visualization design}},
volume = {20},
year = {2014}
}
@article{Netzel2014,
author = {Netzel, R. and Burch, M. and Weiskopf, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
title = {{Comparative eye tracking study on node-link visualizations of trajectories}},
volume = {20},
year = {2014}
}
@article{Palmas2014,
author = {Palmas, G. and Bachynskyi, M. and Oulasvirta, A. and Seidel, H.-P. and Weinkauf, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2359--2368},
title = {{MovExp: A versatile visualization tool for human-computer interaction studies with 3D performance and biomechanical data}},
volume = {20},
year = {2014}
}
@article{Pandey2014a,
author = {Pandey, A. V. and Manivannan, A. and Nov, O. and Satterthwaite, M. and Bertini, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2211--2220},
title = {{The persuasive power of data visualization}},
volume = {20},
year = {2014}
}
@article{Perin2014b,
author = {Perin, C. and Dragicevic, P. and Fekete, J.-D.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2082--2091},
title = {{Revisiting Bertin matrices: New interactions for crafting tabular visualizations}},
volume = {20},
year = {2014}
}
@article{Polk2014,
author = {Polk, T. and Yang, J. and Hu, Y. and Zhao, Y.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2339--2348},
title = {{TenniVis: Visualization for tennis match analysis}},
volume = {20},
year = {2014}
}
@article{Ren2014,
author = {Ren, D. and Hollerer, T. and Yuan, X.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2092--2101},
title = {{iVisDesigner: Expressive interactive design of information visualizations}},
volume = {20},
year = {2014}
}
@article{Rubio-Sanchez2014,
author = {Rubio-Sanchez, M. and Sanchez, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2013--2022},
title = {{Axis calibration for improving data attribute estimation in star coordinates plots}},
volume = {20},
year = {2014}
}
@article{Sadana2014,
author = {Sadana, R. and Major, T. and Dove, A. and Stasko, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {1993--2002},
title = {{OnSet: A visualization technique for large-scale binary set data}},
volume = {20},
year = {2014}
}
@article{Saket2014c,
author = {Saket, B. and Simonetto, P. and Kobourov, S. and Borner, K.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2231--2240},
title = {{Node, node-link, and node-link-group diagrams: An evaluation}},
volume = {20},
year = {2014}
}
@article{Sedlmair2014a,
author = {Sedlmair, M. and Heinzl, C. and Bruckner, S. and Piringer, H. and Moller, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2161--2170},
title = {{Visual parameter space analysis: A conceptual framework}},
volume = {20},
year = {2014}
}
@article{Stolper2014b,
author = {Stolper, C. D. and Kahng, M. and Lin, Z. and Foerster, F. and Goel, A. and Stasko, J. and Chau, Duen Horng},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2320--2328},
title = {{GLO-STIX: Graph-level operations for specifying techniques and interactive eXploration}},
volume = {20},
year = {2014}
}
@article{Stusak2014,
author = {Stusak, S. and Tabard, A. and Sauka, F. and Khot, R. A. and Butz, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2201--2210},
title = {{Activity sculptures: exploring the impact of physical visualizations on running activity}},
volume = {20},
year = {2014}
}
@article{Talbot2014a,
author = {Talbot, J. and Setlur, V. and Anand, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2152--2160},
title = {{Four Experiments on the Perception of Bar Charts}},
volume = {20},
year = {2014}
}
@article{Tennekes2014,
author = {Tennekes, M. and de Jonge, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2072--2081},
title = {{Tree colors: Color schemes for tree-structured data}},
volume = {20},
year = {2014}
}
@article{Turkay2014,
author = {Turkay, C. and Slingsby, A. and Hauser, H. and Wood, J. and Dykes, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2033--2042},
title = {{Attribute signatures: Dynamic visual summaries for analyzing multivariate geographical data}},
volume = {20},
year = {2014}
}
@article{VandenElzen2014a,
author = {van den Elzen, S. and van Wijk, J. J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2310--2319},
title = {{Multivariate network exploration and presentation: From detail to overview via selections and aggregations}},
volume = {20},
year = {2014}
}
@article{VanderCorput2014,
author = {van der Corput, P. and van Wijk, J. J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2301--2309},
title = {{Effects of presentation mode and pace control on performance in image classification}},
volume = {20},
year = {2014}
}
@article{VanGoethem2014,
author = {van Goethem, A. and Reimer, A. and Speckmann, B. and Wood, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2053--2062},
title = {{Stenomaps: Shorthand for shapes}},
volume = {20},
year = {2014}
}
@article{Wood2014,
author = {Wood, J. and Beecham, R. and Dykes, J.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2171--2180},
title = {{Moving beyond sequential design: Reflections on a rich multi-channel approach to data visualization}},
volume = {20},
year = {2014}
}
@article{Zgraggen2014,
author = {Zgraggen, E. and Zeleznik, R. and Drucker, S. M.},
journal = {IEEE Trans. Visualization and Computer Graphics (Proc. InfoVis)},
number = {12},
pages = {2112--2121},
title = {{PanoramicData: Data analysis through pen \& touch}},
volume = {20},
year = {2014}
}
}}}
[[@mattbrehmer's Storify twitter stream|https://storify.com/mattbrehmer/ieeevis15]]
!Sunday
!!Tutorial: ~XData Tools
Joseph Cottam, Peter Wang, Jeff Baumes, Arvind Satyanarayan
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-XDATA.pdf]]
!!Workshop: [[Personal Visualization: Exploring Data in Everyday Life|http://www.vis4me.com/personalvis15/info.html]]
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-PVIS.pdf]]
!!!Collaboration and Learning
!!!!Invited Talk: Collaboration and Learning
Bongshin Lee
!!!!Paper: Sharing Information from Personal Digital Notes using ~Word-Scale Visualizations
Pascal Goffin
!!!!Paper: Personal Visualization for Learning
Jonathan C. Roberts
!!!!Paper: Collaborative Visualizations of Self-impacted data
Pierre Vanhulst
!!!!Paper: Enabling ~Self-Awareness for Knowledge Workers Through Visualization of Instrumented Data
Sidharth Thakur
!!!Reflection and Engagement
!!!!Invited Talk: Reflection and Engagement
Lyn Bartram
!!!!Paper: Are there networks in maps? An experimental visualization of personal movement data
Heike Otten
!!!!Paper: User-driven Expectation Visualization: Opportunities for Personalized Feedback
~Yea-Seul Kim
!!!!Paper: Personal Movie Recommendation Visualization from Rating Streams
Aidong Lu
!!!!Paper: Designing Personal Visualizations for Different People: Lessons from a Study with Elite Soccer Teens
Søren Knudsen
!!!!Paper: Towards a Taxonomy for Evaluating User Engagement in Visualization
Narges Mahyar
!!Workshop: [[businessVis15|http://entsci.gatech.edu/businessvis15/program.html]]
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-business%7Cvis%7C15.pdf]]
!!!Keynote: Stakeholders and Implementers: Exploring Enterprise Visualization Ecosystems
Dr. Alan Keahey (IBM)
!!!Exploratory and Decision Support Tools
!!!!~Multi-Dimensional Comparative Visualization for Patent Landscaping
Kent Wittenburg, Georgiy Pekhteryev
!!!!Sequencing the Enterprise Genome: Interactive Visual Analysis of ~Multi-Dimensional Alliance Activities of Global Enterprises
Rahul C. Basole, Arjun Srinivasan, Timothy Major
!!!Case Studies in Different Business Functions and Industry Domains
!!!!Integrated Interactive Visual Analytics Framework for Supply Network Management
Hyunwoo Park, Marcus Bellamy, Rahul C. Basole
!!!!Incorporating Tabletop Visual Analytics into the Decision Making Process: A Case Study of Retail Banking
Erdem Kaya, Mert Toka, Atilla Bayrak, Burcin Bozkaya, Selim Balcisoy
!!!!Time is of the Essence: Value Stream Mapping a National Health Service (NHS) Emergency Department in the South West of England
Sian ~Joel-Edgar
!!!!~RTP-Vis: Visualizing Ecosystem of Companies and Founders in Research Triangle Area
Sidharth Thakur, Charles Schmitt, Maryann Feldman, Nichola Lowe
!Monday
!!VDS Symposium
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-VDS.pdf]]
!!!Data Science and Visualization for Scientific Discovery
!!!!Invited Talk: Visual Data Science - Advancing Science through Visual Reasoning
Torsten Möller
!!!!Invited Talk: Visualization for Discovery
Jeff Heer
!!!!Invited Talk: Visualization in Public Health
Rumi Chunara
!!!Health and ~Spatio-Temporal Data
!!!!Panel I: Challenges in Visualization for Data Science
Torsten Möller, Jeff Heer, and Rumi Chunara Moderator: Hanspeter Pfister
!!!!Paper: Service Oriented Development of Information Visualization of the Electronic Health Records for Population Data Set
Jaehoon Lee, Thomas Oniki, Nathan Hulse, and Stanley Huff
!!!!Paper: ~RioBusData: Visual Data Analysis of Outlier Buses in Rio de Janeiro
Aline Bessa, Fernando de Mesentier Silva, Rodrigo Frassetto Nogueira, Enrico Bertini, and Juliana Freire
!!!!Paper: Quality of Movement Data: from Data Properties to Problem Detection
Gennady Andrienko, Natalia Andrienko, and Georg Fuchs
!!!Journalism, Sports, and Data Exploration
!!!!Invited Talk: Doing Data Science at News Corp
Rachel Schutt
!!!!Invited Talk: Space, Time, and Skill: Understanding High Performance Sport
Luke Bornn
!!!!Invited Talk: Interactive Online Data Exploration and Analytics
Feifei Li
!!Visualization in Practice
!!!!Paper: ~High-Category Glyphs in Industry
Richard Brath
!!!!Paper: Visual Analytics for Improving Efficiency in Mining Operations
Gilad Saadoun, Peter Bak, and Jonathan Bnayahu
!!!!Paper: Statistical Forecasting in the Energy Sector: Task Analysis and Lessons Learned from Deploying a Dashboard Solution
Thomas Muhlbacher, Clemens Arbesser, and Harald Piringer
!!!!Paper: Analysis and Visualization of Clostridium Difficile Hospital ~In-Ward Transmissions
Margaret Varga, Caroline Varga, and Ben Huston
!!!!Paper: A Novel Distance Measure for Ocean Reconstruction from Sparse Observations Demonstrated on the Atlantic
Markus Kronenberger, Lorraine E. Lisiecki, Christopher Weber, Carlye Peterson, Geoffrey Gebbie, Howard J. Spero, Oliver Kreylos, Bernd Hamann, and Louise H. Kellogg, and Hans Hagen
!Tuesday
!!Keynote
!!!An Evolving Visual Language: Connecting General Audiences to Science through Data Visualization
Donna J. Cox
!!VAST
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-VAST.pdf]]
!!!Temporal Network Data
!!!!Paper: Reducing Snapshots to Points: A Visual Analytics Approach to Dynamic Network Exploration
Stef van den Elzen, Danny Holten, Jorik Blaas, and Jarke J. van Wijk
!!!!Paper: ~MobilityGraphs: Visual Analysis of Mass Mobility Dynamics via ~Spatio-Temporal Graphs and Clustering
Tatiana von Landesberger, Felix Brodkorb, Philipp Roskosch, Natalia Andrienko, Gennady Andrienko, and Andreas Kerren
!!!Platforms of Visual Analytics
!!!!Paper: Personal Visualization and Personal Visual Analytics
Dandan Huang, Melanie Tory, Bon Adriel Aseniero, Lyn Bartram, Scott Bateman, Sheelagh Carpendale, Anthony Tang, Rob Woodbury
!!!!Conference Paper: Collaborative Visual Analysis with ~RCloud
Carlos Scheidegger, Gordon Woodhull, Stephen North, Simon Urbanek
!!~InfoVis
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-InfoVis.pdf]]
!!!Projections
!!!!TVCG: Perception-based Evaluation of Projection Meth- ods for Multidimensional Data Visualization
Ronak Etemadpour, Robson Motta, Jose Gustavo de Souza Paiva, Rosane Minghim, Maria Cristina Ferreira de Oliveira, Lars Linsen
!!!!Paper: Probing Projections: Interaction Techniques for Interpreting Arrangements and Errors of Dimensionality Reduction
Julian Stahnke, Marian Dork, Boris Muller, and Andreas Thom
!!!Networks
!!!!Paper: HOLA: ~Human-like Orthogonal Network Layout
Steve Kieffer, Tim Dwyer, Kim Marriott, and Michael Wybrow
!!!!Paper: ~AmbiguityVis: Visualization of Ambiguity in Graph Layouts
Yong Wang, Qiaomu Shen, Daniel Archambault, Zhiguang Zhou, Min Zhu, Sixiao Yang, and Huamin Qu
!!!!TVCG: Representing Uncertainty in Graph Edges: An Evaluation of Paired Visual Variables
Hua Guo, Jeff Huang, David H. Laidlaw
!!Panel: Color Mapping in VIS: Perspectives on Optimal Solutions
--~Theresa-Marie Rhyne-- (organizer), David Borland, Kenneth Moreland, Bernice Rogowitz, Francesca Samsel, Maureen Stone, Cynthia Brewer
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-colour.pdf]]
!Wednesday
!!~InfoVis
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-InfoVis.pdf]]
!!!Applications
!!!!Paper: Visual Mementos: Reflecting Memories with Personal Data
Alice Thudt, Dominikus Baur, Samuel Huron, and Sheelagh Carpendale
!!!!Paper: Visualization, Selection, and Analysis of Traffic Flows
Roeland Scheepens, Christophe Hurter, Huub van de Wetering, and Jarke J. van Wijk
!!!!Paper: Visually Comparing Weather Features in Forecasts
P. Samuel Quinan and Miriah Meyer
!!!!Paper: Vials: Visualizing Alternative Splicing of Genes
Hendrik Strobelt, Bilal Alsallakh, Joseph Botros, Brant Peterson, Mark Borowsky, Hanspeter Pfister, and Alexander Lex
!!!!Paper: ~TimeSpan: Using Visualization to Explore Temporal ~Multi-dimensional Data of Stroke Patients
Mona Hosseinkhani Loorak, Charles Perin, Noreen Kamal, Michael Hill, and Sheelagh Carpendale
!!!Design Studies and Methodology
!!!!Paper: Sketching Designs using the Five ~Design-Sheet Methodology
Jonathan C. Roberts, Chris Headleand, and Panagiotis D. Ritsos
!!!!TVCG: Bridging Theory with Practice: An Exploratory Study of Visualization Use and Design for Climate Model Comparison
Aritra Dasgupta, Jorge Poco, Yaxing Wei, Robert Cook, Enrico Bertini, Claudio T. Silva
!!!!Paper: Speculative Practises: Utilizing ~InfoVis to Explore Untapped Literary Collections
Uta Hinrichs, Stefania Forlini, and Bridget Moynihan
!!!!Paper: Poemage: Visualizing the Sonic Topology of a Poem
Nina ~McCurdy, Julie Lein, Katharine Coles, and Miriah Meyer
!!!!Paper: Matches, Mismatches, and Methods: ~Multiple-View Workflows for Energy Portfolio Analysis
Matthew Brehmer, Jocelyn Ng, Kevin Tate, and Tamara Munzner
!!VAST
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-VAST.pdf]]
!!!Uncertainty, Correlation, and Causality
!!!!Paper: The Visual Causality Analyst: An Interactive Interface for Causal Reasoning
Jun Wang and Klaus Mueller
!!!!Paper: The Role of Uncertainty, Awareness, and Trust in Visual Analytics
Dominik Sacha, Hansi Senaratne, Bum Chul Kwon, Geoffrey Ellis, and Daniel A. Keim
!!!!Paper: An ~Uncertainty-Aware Approach for Exploratory Microblog Retrieval
Mengchen Liu, Shixia Liu, Xizhou Zhu, Qinying Liao, Furu Wei, and Shimei Pan
!!CG&A
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-CG%26A.pdf]]
!!!Personal Visualization
!!!!Paper: Understanding Digital ~Note-Taking Practice for Visualization
Wesley Willett, Pascal Goffin, Petra Isenberg
!!!!Paper: Eye Tracking for Personal Visual Analytics
Kuno Kurzhals, Daniel Weiskopf
!!!!Paper: Characterizing Visualization Insights from ~Quantified-Selfers’ Personal Data Presentations
Eun Kyoung Choe, Bongshin Lee, m.c. schraefel
!!!!Paper: Engaging with Energy in the Informative Home: Challenges and opportunities for ~eco-feedback
Lyn Bartram
!!!!Paper: Design and Effects of Personal Visualizations
Shimin Wang, Yuzuru Tanahashi, Nick Leaf, ~Kwan-Liu Ma **(presented by Bryan)**
!Thursday
!!~InfoVis
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-InfoVis.pdf]]
!!!Human Reasoning
!!!!Paper: How do People Make Sense of Unfamiliar Visualization?: A Grounded Model of Novice’s Information Visualization Sensemaking
Sukwon Lee, ~Sung-Hee Kim, ~Ya-Hsin Hung, Heidi Lam, Youn-ah Kang, and Ji Soo Yi
!!!!Paper: Learning Visualizations by Analogy: Promoting Visual Literacy through Visualization Morphing
Puripant Ruchikachorn, Klaus Mueller
!!!!Paper: Acquired Codes of Meaning in Data Visualization and Infographics: Beyond Perceptual Primitives
Lydia Byrne, Daniel Angus, and Janet Wiles
!!!!Paper: Beyond Memorability: Visualization Recognition and Recall
Michelle A. Borkin, Zoya Bylinskii, Nam Wook Kim, Constance May Bainbridge, Chelsea S. Yeh, Daniel Borkin, Hanspeter Pfister, and Aude Oliva
!!!!Paper: Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability
Alvitta Ottley, Evan M. Peck, Lane T. Harrison, Daniel Afergan, Caroline Ziemkiewicz, Holly A. Taylor, Paul K. J. Han, and Remco Chang
!!!Interactive Systems
!!!!Paper: Suggested Interactivity: Seeking Perceived Affordances for Information Visualization
Jeremy Boy, Louis Eveillard, Françoise Detienne, and ~Jean-Daniel Fekete
!!!!TVCG: ~VectorLens: Angular Selection of Curves within 2D Dense Visualizations
Maxime Dumas, Michael ~McGuffin, Patrick Chasse
!!!!Paper: Voyager: Exploratory Analysis via Faceted Browsing of Visualization Recommendations
Kanit Wongsuphasawat, Dominik Moritz, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer
!!!!TVCG: ~VisDock: A Toolkit for ~Cross-Cutting Interactions in Visualization
Jungu Choi, Deok Gun Park, Yuet Ling Wong, Eli Raymond Fisher, Niklas Elmqvist
!!!!Paper: Reactive Vega: A Streaming Dataflow Architecture for Declarative Interactive Visualization
Arvind Satyanarayan, Ryan Russell, Jane Hoffswell, and Jeffrey Heer
!!!Techniques
!!!!Paper: Automatic Selection of Partitioning Variables for Small Multiple Displays
Anushka Anand and Justin Talbot
!!!!Paper: A Simple Approach for Boundary Improvement of Euler Diagrams
Paolo Simonetto, Daniel Archambault, and Carlos Scheidegger
!!!!Paper: ~AggreSet: Rich and Scalable Set Exploration using Visualizations of Element Aggregations
M. Adil Yalcın, Niklas Elmqvist, and Benjamin B. Bederson
!!!!TVCG: ~UnTangle Map: Visual Analysis of Probabilistic ~Multi-Label Data
Nan Cao, ~Yu-Ru Lin, David Gotz
!!!!Paper: A Linguistic Approach to Categorical Color Assignment for Data Visualization
Vidya Setlur and Maureen C. Stone
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-InfoVis-Stone.pdf]]
!!Panel: Vis, The Next Generation: Teaching Across the ~Researcher-Practitioner Gap
Marti A. Hearst (organizer), Eytan Adar (organizer), Robert Kosara, Tamara Munzner, Jon Schwabisch, --Ben Shneiderman--
!Friday
!!~InfoVis
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-InfoVis.pdf]]
!!!Time
!!!!Paper: ~TimeNotes: A Study on Effective Chart Visualization and Interaction Techniques for ~Time-Series Data
James Walker, Rita Borgo, and Mark W. Jones
!!!!Paper: Time Curves: Folding Time to Visualize Patterns of Temporal Evolution in Data
Benjamin Bach, Conglei Shi, Nicolas Heulot, Tara Madhyastha, Tom Grabowski, and Pierre Dragicevic
!!!!TVCG: An Efficient Framework for Generating Storyline Visualizations from Streaming Data
Yuzuru Tanahashi, **~Chien-Hsin Hsueh**, ~Kwan-Liu Ma
!!!!TVCG: ~ThemeDelta: Dynamic Segmentations over Temporal Topic Models
Samah Gad, Waqas Javed, Sohaib Ghani, **Niklas Elmqvist**, Tom Ewing, Keith N. Hampton, Naren Ramakrishnan
!!VAST
*[[notes|https://dl.dropboxusercontent.com/u/6397998/VIS2015/VIS-VAST.pdf]]
!!!Visual Analytics of Textual Data (II)
!!!!Paper: ~TimeLineCurator: Interactive Authoring of Visual Timelines from Unstructured Text
Johanna Fulda, Matthew Brehmer, and Tamara Munzner
!!Capstone
!!!Architectures Physical and Digital
Molly Wright Steenson
<!--{{{-->
<div class='toolbar' macro='toolbar [[ToolbarCommands::ViewToolbar]]'></div>
<div class='title' macro='view title'></div>
<div class='subtitle'><span macro='view modifier link'></span>, <span macro='view modified date'></span> (<span macro='message views.wikified.createdPrompt'></span> <span macro='view created date'></span>)</div>
<div class='tagging' macro='tagging'></div>
<div class='tagged' macro='tags'></div>
<div class='viewer' macro='view text wikified'></div>
<div class='tagClear'></div>
<!--}}}-->
!Keynote: //Evaluating Novel Visualizations//
Mary Czerwinski, Microsoft Research
*A survey of lab studies, field studies, longitudinal sensor/logging studies, survey ~MTurk studies
*Lab studies:
**''Data Mountain'': ~3D visualization for tracking browsing bookmarks, included occlusion prevention, spatial audio cues
**Connecting lines to indicate set inclusion (~VisWeek '11): map overlays, alternative to Venn diagrams, sketching study
*Field studies:
**''~ClassSearch'': search classes in middle school classrooms, monitoring and history tracking search progress, visualizing as sequential word / tag clouds
**''Entendre'': ambient lotus-flower (or sun-moon-see-saw) visualization to indicate level of empathy in doctor-patient conversations; hired trained actor as confederate, ~Wizard-of-Oz study
*Sensors, life logging:
**''~AffectAura'': using Kinect, life logging, cameras, app/email/phone logging to track trends in personal affect / emotional state
*also mentioned:
**''~FastStash'' (sp?): increased group awareness of code base edits/ownership among MS development teams
''Questions'':
*''Florian Block'': linking positive evaluation results to tool adoption?
**''~FastStash'' users unwilling to relinquish prototype
*''JS Yi'': evaluation resources for junior students?
**learn by doing MS interns often run their first study during an internship, not a technical contribution
*''Silvia Miksch'': how to measure insight for VA tools?
**the value of sensors (but what is the quality of the insight?)
!Capstone: //Help Me See! Some Thoughts From a Potential User//
Felice C. Frankel, MIT Research Scientist
*(1) help me see (2) help me understand: what do you want me to see? tell me how you think (3) collaborate
*//No Small Matter// e.g. ferrofluid images
*distraction is the worst thing in our work
*responsibility: conveying uncertainty, considering how and what you show
*representations are re-presentations: you are not showing the thing, but a representation of the thing
*physical re-presentations of things with no physicality e.g. electron clouds
*//Visual Strategies//
*rethink colour and its power - it is often misused
*rethinking visual composition: give space, uniform labelling (arrows, lower-case text labels), insets
*tell me how you think: have the viewer participate in your thinking: getting you to share your thinking with me is a way to get engaged
*we like and understand methaphors: we need a library of metaphors
*collaborate: [[Image and Meaning|http://imageandmeaning.com]] [[Graphics to Learn|http://picturingtolearn.org]] - sketching exercises to reveal misconceptions
*Am I saying anything with this information? just because it's data, it doesn't need to be in there
*We don't need cute
*closing question: is any of this relevant to you? yes?
''Comments'':
*contrasts w/ Bloomberg talk on showing everything
''Questions'':
*''Q'': exploration? is the first look critical?
**it depends on how it's represented. should you care about how it's represented from the beginning does that come later after the data is collected? considering how it's represented from the beginning. Give me a number of ways to look at it.
*''G. Shneiderman'' (Leipzig): in visualization, we can show everything. how do we communicate the relevance of a model? To allow different points of view?
**if you approach your data with an already existing model, then you're not going to see anything new. Is this wrong? Give a sense of the forest and the trees: reorient your perspective. CS folks get so engaged in the mathematics of it, they see the model. Change your point of view while you're visualizing it. The way we collect our data is directing the way that we look at it.
*''Hamish Carr'' (Leeds): is this relevant? (Q: milkshake.) (A: happy birthday.) (surrealist answer). An impressionistic, stroboscopic view of someone else's approach. Do you have a systematic approach?
**I'm not an artist, I'm trained in science. This makes a difference. Primarily and foremost I want to communicate the science. I don't want to show you my art. It is not art. It is communication of phenomena. I am also a photographer and I user photographic tools. I use light. It is ultimately about showing the science and showing it sparingly. Simple things are the most difficult to show. It has to be spare and unintimidating. Engage the viewer so they want to know more. Making it accessible in a way gets the viewer to participate. That's my problem with the [Vis] world. You just want to talk to each other. You have to make your work more accessible, to the outside world. It feels very different from how it was 7 years ago. Aesthetics are improving.
*''Hamish Carr'' (Leeds): you've just articulated your philosophy. Is there a 12 step program for recovering bad vis creators, for systematically generating these images.
*''Roy Ruddle'' (Leeds): Re: serendipitous discovery, exploration; these often requires interaction. If you were making a movie, rather than a still photograph, how would you go about creating these? The minimalist approach leads to the tension that relationships are not seen.
**In the book, we discuss interactivity. Journals are heading in this direction. The design of your work should involve outsiders, people from outside your field. I'm not saying that you should reduce your data, but the representations of your data. Show it in different ways.
*''Robert Kosara'' (Tableau): re: example of currents and acidity, molecule of cute bubble. To work against domain conventions (colours, shapes) and expectations and what they're used to seeing.
**we need to stop perpetuating lousy stuff, the Head of Nature was involved in helping to solve these problems.
!~InfoVis Sessions
!!Evaluation and Methodology
//Chair//: Leland Wilkinson
!!!How Capacity Limits of Attention Influence Information Visualization Effectiveness [Best Paper]
!!!!<<cite Haroz2012a bibliography:VisWeek2012>>
Steve Haroz, David Whitney
*intro: visualization primer
*limited spatial window of visual attention vs. feature-based attention
*3 experiments: 8x8 grid, 8 colours - intended to simulate treemap, space-filling display, scatterplot; varied number of unique colours in a display (1-8)
*experiments:
**1st condition: is target present or absent in blocked layout of colours? e.g. finding a particular file type in a directory
***result: consistently difficult w.r.t. time taken as # colours added
**2nd condition: even with random layouts of colours, performance is consistent, though more difficult than blocked colour layout
**3rd condition: looking for unique value but not knowing what colour it will be (akin to searching for something different, an outlier) - serial processing, not parallel: response time increases as colours added
**4th: comparing relative heterogeneity of displays (e.g. 1 vs. 2 colours): 5-6 colours is the maximum number of colours before performance accuracy drops to below chance
**5th condition: same as 4, with random layout, performance drops off earlier after 2-3 colours
*conclusions: capacity for attention is boundary for interactivity
**fewer bins for categorical data or require interactivity to limit the number of bins
**grouping data with a layout (e.g. treemap display)
**e.g. squarified treemaps sorted by area,
''Questions'':
*''JS Yi'': why does 3rd condition have solid line? more colours than 7?
**there is increase
**see Chris Healey paper on colour discriminability (hard to tell after 7-9 colours)
*''Miriah Meyer'': heatmap encodings; continuous colour scale; clusterings to reveal trends; extend to scalar data (in heatmaps)?
**scalar is harder (binning?)
*''Q'': plateau above 50% in exp. 5 vs. drop off point in exp. 6? further experiments to understand phenomenon?
**Jason Fisher (MIT) looking into attention
*''Scott David'' (NASA JPL) - (in movie trailer voice): can static rules extend to dynamic visualizations?
**with motion, same pattern but drop-off occurs earlier
''Notes'':
*well presented, convincing, well-paced, nice slide deck
*''Bernice Rogowitz'': fundamentally flawed because luminance and saturation varied (in addition to hue). (What about effect of size?) Too many confounding factors.
!!!Different Strokes for Different Folks: Visual Presentation Design Between Disciplines
!!!!<<cite Gomez2012b>>
Steven R. Gomez, Radu Jianu, Caroline Ziemkiewicz, Hua Guo, David H. Laidlaw (Brown)
*Meta-presentation on design behaviour in different disciplines (mathematicians, artists, economists, painters); retargeting presentations of information for different audiences
*disciplines: natural sciences, social sciences, formal sciences, humanities
*artifact analysis of presentations from different disciplines: what are the conventions?
**diagrammatic, textual, narrative features in 65 slideshows from 4 disciplines, multiple coders, inter-coder frequency and slideshow length normalization
**inspired by whiteboard study in Walny ~InfoVis 2011 (Visual Thinking in Action)
**sig. effect of discipline (MANOVA), features that occur only in some disciplines
**MDS on slideshow features, discipline pairwise MDS
**Eigenslides, PCA on greyscale slides - mean slides from different dimensions actually look different (e.g. humanities slides tend to have centred images)
*do conventions migrate? controlled study to see if those from some disciplines explain other disciplines with their native design behaviour
**whiteboard explanations: no feature significance; anecdotal evidence suggests disciplinary representations/formalizations carry over
*Conclusions: distributions of features occur in different frequencies in different disciplines - implications for toolbuilding
''Questions'':
*''T. ~Jankun-Kelly'': role of expertise: junior researchers vs. long-term practitioners?
**Authors wanted to collect more data from experienced researchers.
*''T. ~Jankun-Kelly'': MDS results on formal sciences varies. should reclassify?
**e.g. CS/cryptography vs. CS/visualization - highly multidisciplinary fields
*''Q'': generate artificial presentation and showing it to disciplines?
**great idea, implications for automatic generation
!!!Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making
!!!!<<cite Kim2012a>>
~Sung-Hee Kim, Zhihua Dong, Hanjun Xian, Benjavan Upatising, Ji Soo Yi (HIVE lab Purdue)
*comparing sorting interfaces: single-column sort vs. parallel coordinate simultaneous sort with linked highlighting (~SimulSort, Hur et al 12)
*Q: why is ~SimulSort helping? higher decision accuracy, less time
*process tracing method, understanding cog. processes
**eye-tracking study, based on eye-mind hypothesis, looked at fixations and saccades
**heat maps looked different for ~SimulSort and traditional sort
**spatial encoding most effective (Mackinlay '86)
*decision making strategy: task is to select overall best item, requires consideration of more attributes
**explained by compensatory strategies, area-of-interest (AOI) (eye tracking metric)
**AOI: participants don't look at all the attributes, even in ~SimulSort (middle attributes are considered more - peripheral vision?)
*~MTurk study: ~SimulSort is effective for capitalizing on effective browsing, compensatory activity
*Conclusion: complimentary
''Questions'':
*''Hannah Pileggi (GT)'': ~L-R reading habit. Effect of cultures?
''Notes'':
*''Bernice Rogowitz'': flawed because experts may have different scanning patterns,
!!!Design Study Methodology: Reflections from the Trenches and the Stacks [Honorable Mention]
!!!!<<cite Sedlmair2012c>>
Michael Sedlmair (UBC), Miriah Meyer (Utah), Tamara Munzner (UBC)
''Questions'':
*''Enrico Bertini (NYU)'': DS problem: where is the contribution of the design study research?
**reflection stage: what is the contribution? confirming, rejecting, proposing guidelines for the community, transferability, BELIV paper
!!!Graphical Tests for Power Comparison of Competing Designs ᡦlt;<cite Hofmann2012a>>
Heike Hofmann, Lendie Follett, Mahbubul Majumder, Dianne Cook (Iowa State)
*visual inference(Wickham 10, Buja 09), lineups / decoy plots
**there is signal in the data, people spot it with an accuracy that is far less than chance (p < 10^^-4^^)
*methodology for ~A-B comparative lineup studies; what affects power?
*logistical regression for individual traits: case study of airport efficiency and wind patterns (FAA database)
*experimental design: two alternative designs for displaying wind patterns and airport efficiency (polar coordinates vs. euclidean layout), generating 192 datasets, plots, included these in lineups; deployed on ~MTurk, 958 evaluation w/ 100 participants, attempted to weed out bad data, collected confidence ratings
**results: euclidean design sig. superior to polar charts; helper lines helped with confidence, but not in terms of performance; shifts in polar direction (0ะ簡ffected polar layout, but not euclidean layout
*Conclusion: lineups useful for comparing techniques, define a power function
*2nd case study in paper
Questions:
*Polar charts: no area correction; wanted to emphasize areas
!!Graphs and Networks
//Chair//: Nathalie Henry Riche
!!!Memorability of Visual Features in Network Diagrams
!!!!<<cite Marriott2012>>
Kim Marriott, Helen Purchase, Michael Wybrow, Cagatay Goncu
*reporting experiments on the effect of symmetry, node alignment, horizontal/vertical features on recall (redrawing) of node-link diagrams (compared to control)
*collinearity, orthogonality, symmetry are preserved in recalled drawings of node-link diagrams (horizontal and vertical features not recalled well, node-alignment and parallel lines not preserved
''Notes'':
*too much text on slide, poor readability from back of room
''Questions'':
*''Tim Dwyer'': a different type of graph memorability studies; how is this memorability useful in analysis tasks?
**not sure. it might inform the way information is highlighted, parts of the network to preserve in terms of a mental map
*''Nathalie Riche'': size of networks? Do results scale?
**suspects some of them will, symmetry most likely to scale. Others not certain.
*''Q'': would you use sketching software again?
**pros/cons to this. It would be easier to label rather than redraw/sketch - but didn't want to associate any other semantic information with nodes.
!!Representation and Perception
//Chair//:Enrico Bertini
!!!Visual Semiotics & Uncertainty Visualization: An Empirical Study [Honorable Mention]
!!!!<<cite MacEachren2012>>
Alan M. ~MacEachren, Robert E. Roth, James O宬 Bonan Li, Derek Swingley, Mark Gahegan (PSU)
*abstract and iconic representations for representing spatial uncertainty; accuracy, perception, trustworthiness
*experiment: measuring intuitiveness of a sign (referent, interpretant/meaning)
**results: attributes of intuitiveness: fuzziness, location, value highly intuitive, arrangement and size
*integration of intuitive symbols into a map-like display: 2nd experiment compares maps of varying uncertainty levels (student users)
**iconic symbols didn't have strong perceptual ordering
*conclusions: implications of scale, size for abstract and iconic symbols; what is the role of background/foreground effects? iconic inappropriate for continuous variables
''Notes'':
*too much text / data on slides
*See Ron Rensink's PSYC 579 course notes
*reminds me of Scott ~McCloud's understanding comics
''Questions'':
*''Chirs Johnson'': what are the gradations of fuzziness?
**many symbols don't extend to more than 3 gradations. Likely that more than 2 categories of uncertainty is not feasible.
*''Robert Kosara'': blur and fuzzyness: sometimes we don't need to display all levels of uncertainty, all levels of the data
!!!Comparing Clusterings Using Bertin䥡
!!!!<<cite Pilhofer2012>>
Alexander Pilh적lexander Gribov, Antony Unwin (U. Augsburg)
*comparing clustering algorithms - categorical and classification data: how do the clusterings agree?
*arranging data along pseudo-diagonals / Bertin's Classification Criterion (BCC)
*re-arranging a confusion matrix, a circos plot, a parallel coordinates plot to better display clusterings (generalizes to other visualizations, multivariate extensions)
*BCC extends to multivariate parallel coordinates
*Possible to evaluate a plot with the Bertin Classification Index (BCI) - normalized BCC
*efficient algorithm (greedy) (shows pseudocode)
*can be extended to hierarchical clustering, automated clustering
''Notes'':
*heavy with equations, too dense for speed/heterogenous audience
*Math! shows equation, "I will not go into detail here"
*Confusion matrix no longer symmetric? Isn't this a problem?
''Questions'':
*''Enrico Bertini'': what are the other approaches? How do they differ?
**e.g. barycentric coordinates, but doesn't extend to multiple dimensions. Others in Parallel coordinates and circos plots that minimize crossings, butၳsessing the Effect of Visualizations on Bayesian Reasoning Through Crowdsourcing [Honorable Mention] ᡦlt;<cite Micallef2012>>
Luana Micallef (Kent), Pierre Dragicevic, ~Jean-Daniel Fekete (INRIA Aviz)
*Example of woman with positive mammography test result (P misdiagnosis 10%), what is the probability that she actually has cancer? (P Cancer in population 1%)
**many doctors get this wrong
*many representations are intended for specialized students, not area-proportionate
*crowdsourcing study to explore 6 alternatives to visual representations for this (163 workers)
**subjects confident but often wrong, no difference in error amounts b/w alternative visualizations, no visualization control group
**89% found diagram helpful, but doubted it?
*solution: change the text prompt (task), 2nd study (460 workers)
**most effective textual representation is to provide no numbers in the instruction, numbers throw us off; least amount of errors
*simply adding a visualization to a text doesn't help, but a diagram can be helpful if numbers are removed from texts
''Notes'':
*~EulerGlyphs Bayesian reasoning vis
*claim that ~MTurk is more like the "real world"
*word cloud closing slide!
*explanation for removing numbers?
''Questions'':
*''Caroline Ziemciewitz'': visual/numerical/spatial literacy effects?
**no sig. effects; deferred to some thesis
*''Q'': ~MTurk replication? Effect of culture (~MTurkers from India)᭢led) (out of time)
!!TVCG on Information Visualization
//Chair//: Stephen North
!!!Empirical Studies in Information Visualization: Seven Scenarios
!!!!<<cite Lam2011>>
Heidi Lam (Google), Enrico Bertini (Konstanz), Petra Isenberg (INRIA), Catherine Plaisant, Sheelagh Carpendale (Calgary)
*2 scenarios picked: evaluating visual analysis process and evaluating user performance (objective)
*superimposed high-level tasks on sensemaking loop: data exploration (1-7) > knowledge discovery > hypothesis generation > decision making (16)
*Mackinlay '07 - usage stats in Tableau
*bad vis for trends in visualization: angled x-axis labels, no y-axis label
*Why do we see fewer process evaluations? We don't know. Qualitative data gathering, long process - is our community not ready to publish this work?
*paper available at [[http://beliv.org/wiki]] - call to add new evaluation scenarios to the wiki
*typos!
''Questions'':
*''Steven North'': how are evaluations added to the wiki? what if evaluations have multiple goals?
**guidelines for adding : a good methodology (that works), methodological details. Multiple goals need to both be described sufficiently.
!!!Enhanced Spatial Stability with Hilbert and Moore Treemaps
!!!!<<cite Tak2012>>
Susanne Tak (TNO, Netherlands), Andy Cockburn (U Canterbury)
*treemaps: slice and dice, squarified, spiral
*metrics of treemaps: aspect ratio, shading, readability, continuity, distance change (for item movement on updates), location drift metric (centre of gravity for each item)
*experiment: item finding across updates - showed that location drift metric predicts experiment
*Hilbert curves / Moore curves perform on these stability metrics almost as well as slice-and-dice treemaps, but aesthetics are stronger
''Notes'':
*run under time
*re: math details: see the paper.
''Questions'':
*''Q'': Jigsaw treemaps?
**these couldn't be used as they're not necessarily square.
*''Q'': algorithm?
**not a mathematician / computer scientist
!!!~LineAO - Improved ~Three-Dimensional Line Rendering
!!!!<<cite Eichelbaum2012>>
Sebastian Eichelbaum, Mario Hlawitschka, Gerik Scheuermann (Leipzig)
*ambient occlusion, shadows, illumination, line bundling, local and global occluders
*~OpenWalnut - tool/code released online
''Notes'':
*~SciVis topic?
*Math! and pseudocode! and code! (full screen) ("trust me")
''Questions'':
*''Q'': resolution? ~720p HD ready
*''Tobias Isenberg'': try dense particle datasets
*''Steven North'': domain experts opinions?
**yes, submitting to domain-specific journals (e.g. fluid motion journal)
*''Q'': weighting function?
**(more math!)
!!!Using Patterns to Encode Color Information for Dichromats
!!!!<<cite Sajadi2012>>
Behzad Sajadi, Aditi Majumder (UC Irvine), Manuel M. Oliveira, RosǮ Schneider (UFRGS, Brazil), Ramesh Raskar (MIT Media Lab)
*overlays for encoding missing colour channel, content independent, interactive
*mapping colours to within dichromats' visual space, but preserving visual distances, without being content-dependent
*adjust contrast of pattern overlay to reduce visual disturbance
*also applies to heatmap visualizations (not w/ lines but points), discrete and continuous scales
*user study with 6 deuteranopes, 5 protanopes
''Notes'':
*3D pie charts! rainbow colour maps!
*does this work for non-area / visualizations?
*result bar charts on 3D bar charts with red and green bars!
*"in futureﴻ
''Questions'':
*''Tobias Isenberg'': still amazed that so many vis applications don't check for colour blindness
!!!Toward Visualization for Games: Theory, Design Space, and Patterns
!!!!<<cite Bowman2012>>
Brian Bowman, Niklas Elmqvist (Purdue), T.J. ~Jankun-Kelly (Mississippi State)
*"I am a visualization researcher, but I live a double life. I play video games." How do visualization and games interact?
*so much user data generated. Every game you play is a source of data
*this is not a gamification talk
*vis in games framework: purpose, audience, usage, complexity, immersion/integration
*examples: static trees in Civ 5, dynamic bar charts in ~SimCity 4, retrospective heatmaps of kills/places killed in Halo Reach, prospective usage, heads-up displays (continuous info)
*value for games, vis research: huge user population, visual literacy / spatial literacy higher among gamers
''Notes'':
*will be tweeting his own talk as he gives it
*"ludos phasmis" (ghosting in time trials in Mirror's Edge)
''Questions'':
*''Enrico Bertini'': did you discover anything that we can apply to visualization not related to games? something in the framework that was unexpected?
**30+ games evaluated, no novel vis; what was most surprising was the data, size of data
*''Steven North'': gamers have higher spatial literacy, but visualizations tend to be basic. Is there an opportunity here?
**~SimCity has had choropleth maps since '89, and now it's in 3D
*''Q'': custom user ~WoW sHUD.
!!Sketching and Designing Visualizations
//Chair//: Caroline Ziemkiewicz
!!!Sketchy Rendering for Information Visualization ᡦlt;<cite Wood2012>>
Jo Wood, Petra Isenberg, Tobias Isenberg, Jason Dykes, Nadia Boukhelifa, Aidan Slingsby (~giCentre)
*digital prototyping (Lloyd Dykes ~InfoVis '11)
*on XKCD: effort adds to impact, authorship, narrative
*using sketchiness as an information-carrying channel (e.g. uncertainty)
*measured vs. perceived sketchiness: poor correlation but still some structure, the 50%, 80%, and 90% lines do stand out; individuals tend to be internally consistent
*potential for sketchy ~InfoVis: carries uncertainty, variability: compared sketchy vs. non-sketchy pie chart comparisons
*challenging preconceptions w/ interactive sketchy visualizations
''Notes'':
*simple, aesthetically pleasing slides (font used: clare hand)
*[[open source implementation|http://gicentre.org/handy]] (processing toolkit)
*see also: [[Automating xkcd Diagrams: Transforming Serious to Funny|http://blog.wolfram.com/2012/10/05/automating-xkcd-diagrams-transforming-serious-to-funny/]]
''Questions'':
*''Q'': skeptical about implications for encoding attribtutes in ~InfoVis. user study with interactive?
**positive engagement in initial studies, challenged preconceptions
*''Robert Kosara'': (1) work back in mid-80s in CAD programs - sketchy renderings in CAD software - intended to use this as a prototyping software. (2) porting from Processing to D3 so it can run in the broswer?
**we'll ask the community to do this, it's open source
*''Q'': perceptual testing? accuracy and perceived values?
**haven't investigated this yet.
!!!An Empirical Study on Using Visual Embellishments in Visualization
!!!!<<cite Borgo2012>>
Rita Borgo, Alfie ~Abdul-Rahman, Farhan Mohamed, Philip W. Grant, Irene Reppa, Luciano Floridi, Min Chen
*visual embellishments have counterparts in figures of speech: compactness, vividness, inexpressibility; these are analogous to cognitive processes: memorization, visual search, concept grasping
*visual embellishments has mixed effects - good for WM, trend effect for LTM, bad for visual search, contextually dependent for concept grasp, sensitive to style
''Notes'':
*brain dump!
*experimental design is confusing. what is the ecological validity of these tasks? who are the users?
*so much text, abbreviations, and result figures on slides
*speed (response time) and accuracy
*"sorry sorry sorry" on slide issues
*too many low-level methodological details
*results overload, overwhelming - who is following this? needs a high level takeaway - can't keep track of all these abbreviations and hypotheses
*more appropriate for perceptual psychology / cogsci venue
*what were the stimuli set? the dataset? distractor task?
''Questions'':
*''Caroline Ziemkiewitz'': chartjunk helps?
**maybe for novice / nonspecialist users - they provide redundancy - it only helps in some contexts (⬥s)
*''Q'': rhetorical figures, visual decoration: embellishment is for the benefit of the author, not the audience
**in ppt, it can be used in both ways
!!!Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty ᡦlt;<cite Boukhelifa2012>>
Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, ~Jean-Daniel Fekete (INRIA)
*"very well in par with Jo Wood's talk"
*sketchiness as a depiction of qual. uncertainty: intuitiveness, effectiveness, preference
*trad. uncertainty: blur, grayscale, dashed lines
*7 ~MTurk studies, avoiding random clickers, $0.34 per trial: comparing sketchiness to blur for abstract and spatial representations, primed (multiple choice vs. open-ended responses
*results indicated that sketchiness is not intuitive, less so for spatial - but neither is blur! both are poor for conveying uncertainty in spatial representations
*sketchiness for value retrieval is very bad (worse than blur, dashed line, grayscale), but people can discriminate 3-4 levels of sketchiness (they can be ranked to within 90%)
*use sketchiness for ordinal scales
*sketchiness was least preferred among Turkers, because of its informality (which is not necessarily a bad thing) (dashing was most preferred)
*however, if we want informality (e.g. Jo Wood's work), we should use sketchiness
''Notes'':
*how can you define maximum sketchiness?
*7 experiments. incremental. like good cognitive psychology experiments.
''Questions'':
*''Robert Kosara'': sketchiness is not a full visual variable: similar to his work on blur. You had a very small amount of uncertainty in your images. Perhaps with larger amounts of uncertainty, blur overlaps and becomes confused.
*''G. Ramos'' (MSR): uncertainty of nodes and edges in spatial
*''Jeff Heer'': sketchiness that scales? implications for clutter?
**to be addressed in future work. sketchiness shouldn't be distracting or for variables that are not too big.
!!!Understanding Pen and Touch Interaction for Data Exploration on Interactive Whiteboards
!!!!<<cite Walny2012>>
Jagoda Walny, Bongshin Lee, Paul Johns, Nathalie Henry Riche, Sheelagh Carpendale (MSR, Calgary)
*everyday physicality + underlying data
*~SketchInsight (~SketchVis)
*WYDIWYG (D = draw) - minimize mode errors, good defaults, autocomplete
*~Wizard-of-Oz evaluation aimed to discover the right interactions
*instead, interactions can be learnable, knowledge transfer from the system, the real world, acknowledge preconceptions from ~WIMPs
*feedback is important: "the tyranny of the blank screen"
*when to use pen vs. when to use touch? a division of labour, mostly transferred from existing interactions
''Notes'':
*references ~NapkinVis! (~InfoVis '10 poster)
*speaks w/ hands, reading speaker notes
*totally open-ended task? what were instructions?
''Questions'':
*''Q'': build a working system?
**ongoing work! the next step of design
*''Chad'' (Georgia Tech): where to start? data or frames? more complex vis?
!!Education and Popular Applications
//Chair//: Danyel Fisher
!!!The ~DeepTree Exhibit: Visualizing the Tree of Life to Facilitate Informal Learning
!!!!<<cite Block2012>>
Florian Block, Michael S. Horn, Brenda Caldwell Phillips, Judy Diamond, E. Margaret Evans, Chia Shen
*design study, many stakeholders: users not domain experts - designing for rather than designing with
*evolutionary biology exhibit: learning goals, effectiveness may take months
*learning > design goals > RITE (rapid iterative testing and evaluation) loop: observation with museum visitors and feedback from other stakeholders, evaluation by cognitive psychologists and learning psychologists
*heterogenous data sources > deep tree database
*conceptual and cognitive difficulties in reading large trees
*fractal layout w/ deep tree (many levels)
*self revealing instructions
*relating and comparing items in the tree, allowing for collaborative multi-user input, requires some social negotiation
*animation and a large dataset provoke an affective response, a sense of awe of the vastness of bioversity
''Notes'':
*methods/methodology ingredients/recipes: takeaway (Chinese takeout metaphor)
*other components as spices: data is a spice? mixing metaphors?
*"recepies"
*conceptual not interrelated w/ cognitive
*desert fog zoom issue
''Questions'':
*''Tobias Isenberg'': where can I buy it? all species at top of the tree, but some species die out and should be at a lower level
**time is a categorical variable; it was a qualitative decision, for simplicity
**in textbooks, all species are represented at the same level
*''Q'': overview and detail? a "you are here" panel?
**for the same reason as above (simplicity), an overview is not as much value because quickly it converges on a single point
*''Tim Lebo'' (RPI): gathering the images?
**most crawled from web API, some error correction (10%)
*''Tim Lebo'' (RPI): logging interactions?
**a semantic log, not low-level input
!!!Living Liquid: Design and Evaluation of an Exploratory Visualization Tool for Museum Visitors ᡦlt;<cite Ma2012>>
Joyce Ma, Isaac Liao, ~Kwan-Liu Ma, Jennifer Frazier
*Exploratorium in SF - exhibit on plankton
*WIP at the time of paper submission, 7 months of development since
*heterogenous datasets, iterative design
*can visitors associate plankton types across zoom levels?
*observed and recorded visitors' questions and comments - did participants understand relationship b/w plankton types and environmental factors?
*more guidance is needed
*current status, zooming lens metaphor, multiple users, zooming w/ physical magnifying lens widget; info buttons
''Notes'':
*obvious linking from focus and context regions?
*clean your video's audio!
''Questions'':
*''Q'': members of team? who was most critical? time factors?
**team: diverse 4-5 team members (part time graphic designer included), weekly meetings, ongoing for at least a year, some periods more active than others; involved w/ Darwin project, another multidisciplinary team
*''Maureen Stone'' (Tableau): why the physical fiducial lenses? multi-touch focus and context was sufficient?
**museum context, joy of physical interaction (acknowledges that these break, get lost, need to be replaced)
!!!Visualizing Student Histories Using Clustering and Composition
!!!!<<cite Trimm2012>>
David Trimm, Penny Rheingans, Marie desJardins (UMBC)
*time-series data of student performance (GPA), how they advanced through course levels (1st year > 4th year level courses)
*design choices: trajectories, opacities, colour weaving: accumulation: localized input > symmetric core transformation or asymmetric core transformation (the former preferred for educational data
*HSV each encoding another value (colour weaving)
*logical groupings (e.g. students who had an A in a certain course) rather than clustering, but reintroduced clustering of the logical groupings
*future work: projection pursuit, interactivity, overview zoom/filter details-on-demand
''Notes'':
*red-green colour scale, results in mess
*abandoned rainbow colour scale later
*MS: does this make any sense?
*difficult to understand the visual encodings, visualization pipeline, the educators' task(s)
*colour weaving?
*odd choices of other domains? soccer ball trajectories?
''Questions'':
*''Steven Drucker'': looks like static images. FW in interactivity? how long to generate images?
**adding interactivity in FW. images generated quickly. code hasn't been optimized yet, clustering was inefficient. You can optimize the code, use the GPU, interactivity can be supported, interactivity is much more "intuitive"
!!!~SnapShot: Visualization to Propel Ice Hockey Analytics
!!!!<<cite Pileggi2012>>
Hannah Pileggi (@achpea), Charles D. Stolper (@chadstolper) , J. Michael Boyle, John T. Stasko (Georgia Tech)
*competitive advantage in analytics (e.g. Moneyball)
*many hockey analytics apps are for the casual fan: no serious data mining / VA
*tool to support hypotheses generation and validation, comparisons
*ultimate goal is communication of results, business intelligence (e.g. every time a team loses, the coaching staff wants findings data)
*heatmaps, small multiple displays
*use cases w/ 3 NHL analysts e.g. hypotheses that shots were shorter at Madison Square Gardens, where home/away teams take their shots
''Notes'':
*paired presentation (HP and CDS), like a sports talk show? tradeoff back and forth slide-by-slide, scripted.
*implemented in D3
*sports analytics course at GT
*why 3 heatmaps types?
*comparisons made across browser tabs - issues for working memory - requires interaction
*wanted something new and creative
''Questions'':
*''Danyel Fisher'': re: analysts' belief that distance was important, but this wasn't shown. Did analysts pause to rethink their hypotheses?
**No. the distance along those 45ᮤs was what was important. Radial heatmap used for drilling down
*''Q'': what about individual player data?
**you can, but many analysts care about overview.
*''Q'': applying other sports?
**yes. some design decisions were based on hockey, but it can predominantly apply to other team sports
**''@jsndyks'': Agreed - to what extent are analysts constrained by expectations of possibilities in their approach to data?
!!Special Session: Tales from the Trenches
//Chair//: Frank van Ham
!!!Game Player Visualization based on Telemetry and Other Data Sources
Jon Hopson (JH, Bungie)
*making the data look like it integrates with the game design
*plotting kill data on heatmaps to redesign maps
!!!Tableau Public Usage Patterns
Ellie Fields (EF, @eleanorpd) (Tableau Public)
*J. Mackinlay has been studying usage of Tableau public
*21K visualization builders (many journalists), 74M impressions b/w '10-'12
*cohort analysis reveals high attrition rate of users: why?
**need to relearn the tool after months of non-use
**a problem with usability? J. Mackinlay investigating
**now offering more online training, more resources in the product
*looking at conversion rates from Tableau public to desktop (but usage data only collected from desktop version, corporate installations)
*Tableau loves interactivity, but there's still a lot of text, non-interactive visualizations, stories around the visualization, pasting static visualizations into emails, monthly reports
!!!Data Visualizations for ~Non-Profits
Kim Rees (KR, @krees, @periscopic, Portland OR)
*living, breathing data (e.g. polar bear populations, salmon spawning, cancer deaths) - easy to get wrapped up by the subject matter, difficult to check emotions at the door, wanting to scream their message from the rooftops - however non-profits want to maintain non-partisanship, low/no bias
*@John Schwabish (CBO), "sorry John, but your data is boring" (but the communication of the data is important - e.g. food stamps)
*examples: geospatial polar bear population visualization, interactive non-partisan matching political candidates to personal preferences (like match.com), bean plots for grant/funding distributions (though they found the visual encoding to be misunderstood by laypersons, reduced confidence, so they switched to heatmaps/matrix views), salmon population
*they use Tableau a lot
*the F word: feelings emotion in non-profit work, often heartbreaking work
!!!Global Burden of Disease: Data and Visualizations
Peter Speyer (PS, @peterspeyer) (Institute for Health Metrics and Evaluation, U. Washington)
*data related to disease, health loss, morbidity, funded by BMGF
*visualizing and comparing uncertainty re: mortality/disease across different countries, pointing out outliers, based on regional estimates if local data is missing
*line charts, treemaps, global scale / local scale: IHME|GBD
*save researchers time searching printouts, tables of data
Discussion:
*''Q'': @PS: small number of visualizations?
**''PS'': you see the entirety of the disease burden, at higher levels of detail you'll only see broad categories of total 290 diseases
*''David Sprague'': each of these visualizations aimed at persuading? how important was interactivity for persuasion?
**''KR'': we do both static infographics and interactive data visualizations; people want to see different things. The political piece required interaction
**''PS'': give them the controls so they can explore, but it's a balance b/w that and telling a story
**''EF'': let people tell their own stories; however, an issue of credibility: if you offer visualization, you give control and explanatory power: simple tooltips can be very powerful. Interactivity also keeps people longer on the site.
**''JH'': an issue ownership.
*''Q'': failure cases?
**''KR'': one client on interaction "can we just get a drop-down menu"? A chasm b/w understandings of interaction. How to appease so many different people. User testing a broad problem: where to start?
**''JH'': designers that say "yeahᴧs //about// what I expected". No element of surprise. It's worse than misunderstanding. There will be no impact.
*''EF'': working w/ a major news corporation - they wanted to strip out analytic activity. They had publishing control. A lot of the stories you could tell were lost. The purpose of visualization is not simply beauty. Analytical usefulness is the most important thing, but aesthetics can get in the door.
*''JS Yi'': @EF acceptance of different graph types?
**''EF'': J Mackinlay has researched this. Scatterplots and heatmaps are hard to get. Maps are easy, even when distorted. Treemaps, bubblemaps, word clouds can also draw people in. Line and bar charts are easy to understand. Tooltips and large legends are important.
*''Robert Kosara'': what about the reaction of "talk is cheap, I know you're trying to play me so I'll move on". However when you see numbers, there's more seriousness to it, there's a gravitas to them.
**''KR'': people see numbers as more valid than words. They trust it more than a story. If users can see that in cold hard facts, it can be more compelling. People interact more with the visualization than the static content on sites. Visualizations are shared on FB, twitter.
**''PS'': on one hand, numbers are robust. Talking with journalists, if you have more than one number in an article, people lose interest.
*''Q'': pet peeves in visualization work?
**''KR'': there are many pet peeves, here at ~VisWeek and in their own work. Any work that's too elaborate is a pet peeve. Simplify. People are overwhelmed easily. Maintaining a respect for the users. Elegant, succinct, takeaways are available.
*''Hannah Pileggi'' (Georgia Tech): what would you like from Vis research?
**''EF'': ~InfoVis is still a young science. Just keep going. Help us to understand how people consume information better. We need more tools for the people. Re: the comment of one number in an article, people aren't stupid, but nevertheless, the one number in an article issue holds.
**''JH'': more analysis of unique video game datasets
**''PS'': we need more publicly available data out there. Creative use of that data.
**''KR'': efficacy of particular graphs. Not the pie chart vs. bar chart debate, but. Innovation. e.g. Philip Glass compendium, the [[ibm glass engine|http://philipglass.com/glassengine/#]] (enables deep navigation of the music of Philip Glass (not really visualization), beautiful ways to represent data, one screen, accessible. Martin Wattenberg's work. Elegant, simple, seamless. I see snippets of it here and there at ~VisWeek. The academic rigour may be stifling innovation. Academics have time to explore and innovate, while industry is constrained, on timelines. Draw out from whatever datasets inspire you. We want those ideas.
*''Q'': @JH mining forum data and dealing with trolling?
**we don't mine that, extremely self-selected, very different hyper-extreme end of players. These users are canary-in-the-coal-mine, but the forum data itself is not used. Instance of forum user who mined play data to discover skew and bias in the game.
!VAST Sessions
!!Text and Categorical Data Analysis
//Chair//: David S. Ebert
!!!Visual Classifier Training for Text Document Retrieval [TVCG]
!!!!<<cite Heimerl2012>>
Florian Heimerl, Steffen Koch, Harald Bosch, Thomas Ertl (Stuttgart)
*Active Learning to bootstrap the classifier; deciding whether a document is relevant/non-relevant/rejected
*User-driven method: decision boundary b/w relevant and unrelevant, displays classified and unclassified data w/ grey and white pts. Lens tool to display keywords, selection method.
*Evaluation: used Reuters news wires, ~VisWeek abstracts
**results: visual feedback preferred, analysts need training, but not necessarily ML training
''Notes'':
*what was the evaluation task? unclear.
*what is visual method? how does it work along with the user-driven method?
*"um" every 3rd word - distracting
''Questions'':
*''Q'': What is similarity metric in 2D?
**Used PCA (first 2 principal components?) - classification uncertainty as other axis?
!!!The Deshredder: A Visual Analytic Approach to Reconstructing Shredded Documents
!!!!<<cite Butler2012>>
Patrick Butler, Prithwish Chakraborty, Naren Ramakrishnan
*O(N!) problem, not a jigsaw problem, sparse dataset, not all shredders equal, alignment is difficult; fully automated computer vision approach is not sufficient
*approaching the document shreds manually, crowdsourcing (no ecological validity / realistic - shredded documents are classified); document reconstruction can take up to 600 man hours
*assumes that shredded strips are rectangular (not always true)
*time series analysis, Chamfer distance distribution
*constraints: orientation or shreds, unmatched edges, lack of useful features
*expert study evaluation: matching performance
*FW: SIFT, SURF
''Notes'':
*Text too small, not speaking into mic, turning away often, can't read plots, see animations
*Unclear how time series plots are used for non-temporal, but spatial pattern
*session well behind time
*blah blah Chamfer equations - too quick or not necessary? something about constraint propagation衴 did experts have expertise in? What was ground truth?
''Questions'':
*''Q'': Where are results? What does a reconstructed document look like?
!!~Visual-Computational Analysis of Multivariate Data
//Chair//: ~Jean-Daniel Fekete
!!!Subspace Search and Visualization to Make Sense of Alternative Clusterings in ~High-Dimensional Data
!!!!<<cite Tatu2012>>
Andrada Tatu, Fabian Maa鮥s F⬠Enrico Bertini, Tobias Schreck, Thomas Seidl, Daniel Keim (U. Konstanz)
*alternative clusterings in HDD
*subspace search, not subspace clustering (surfing): finding the subspace dimensions, uses MDS / PCS / other DR 2D projections
*culling redundancy in the subspaces, grouping subspaces according to similar subspaces: (1) which dimensions do they contain? (2) data topology to hierarchically cluster subspaces (Wards' min variance method)
*confusion matrix of similar and dissimilar dimensions x data topology similar x dissimilar: if they agree, there is redundancy and complementarity of subspaces, also filters out dominant dimensions and confirmatory subspaces - complementary subspaces most interesting
''Notes'':
*screen resolution fubar,
*video demo playing too fast, can't read text
*what is a confirmatory subspace?
*cutoff in terms of time, no questions
!!!~Just-in-Time Annotation of Clusters, Outliers, and Trends in Point- Based Data Visualizations
!!!!<<cite Kandogan2012>>
Eser Kandogan (IBM)
*overlaying plots with external contextual information: e.g. scatterplots of gas price vs. presidential approval rating, overlaid with semantics (Gulf war, '79 energy crisis)
*joining datasets; transformations not yet implemented
*use the ink on the screen to identify visual features (grid-based approach - - susceptible to issues of grid-cell size), then extract the semantics
*suggest semantics for visual features, not specific about accuracy
*how to annotate? several alternatives presented
*how to interact? brushing, more?
*performance: running time scales to >6K points, >20 dimensions
''Notes'':
*Demo of Interstar: scatterplot about cars
*why are categories overlaid on the data with categories? little interaction feedback; strechable canvas; dimensions are subtractive - how to predict which remaining dimensions overlay?
*loves X is峣ription of X
*skipped 4-5 slides - mathematical feature detection details
*no user eval (but a novel interaction experience?) - who are the users? what tasks does it support?
''Questions'':
*''Joseph Cottam'': what are the metrics other than those for cluster detection? what other features are being detected?
**trends: line fitting, outliers: other stat. methods. Those 3 things
*''Q'': will codebase be available? ''A'': maybe.
*''Mike Gleicher'': (1) what were you using to move points as canvas was dragged? (2) what if there are no clusters?
**(1) star coordinates (sp?); (2) ???
!!Sensemaking and Collaboration
//Chair//: Niklas Elmqvist
!!!An ~Affordance-Based Framework for Human Computation and ~Human-Computer Collaboration [TVCG] ᡦlt;<cite Crouser2012>>
R. Jordan Crouser, Remco Chang (Tufts): [[slides|http://goo.gl/ujLGC]]
*meta-review of human-computer-collaboration (HCC)
*curated 49 papers / 1,300 from various HCI/Vis venues
*framework on affordances: affordances b/w agents and machines: computers rely on humans to provide visual perception, visuospatial thinking (e.g. Fold It), audio-linguistic ability, sociocultural awareness, creativity, domain knowledge; humans rely on machines for large scale data manipulation, collection and storage of data, efficient data movement, bias free analysis; HCC systems involve multiple affordance combinations; many systems leverage the same affordance sets: a possibility to compare alternative systems
*examples: Fold It, ~PatVis (Koch '09)
*guidance to leverage affordances, a framework for evaluation, prevent overload: we need to quantify affordances and workload
*Shahaf / Amir '07: human-assisted turing machine
''Notes'':
*relationship b/w affordances and tasks; (1:1) (N:1) (N:N)?
*contribution: common language (a list?)
*cite for task paper
''Questions'':
*''Q'' why not citing work related to automated feedback algorithm for annotating image?
**Not aware of this work.
*''Tim Lebo'' (RPI): what is the common language? Is it a task list?
**language: affordance slides; not tasks, specific affordances, a set of skills that can be applied to multiple tasks
*''Eli Brown'' (Tufts): where are interesting spaces in affordance list?
**some interactions of affordances are not explored e.g. sociocultural knowledge and large scale computation
!!!Examining the Use of a Visual Analytics System for Sensemaking Tasks: Case Studies with Domain Experts [TVCG] ᡦlt;<cite Kang2012>>
Youn-ah Kang, John T. Stasko (Georgia Tech)
*evaluating use of Jigsaw by analysts
*users: multiple intelligence/business analysts, academics
**users had unique goals!
**business analyst found evidence of financial fraud after analyzing ~100K emails
*user tasks: relationships, search/compare, understand, communicate / share
*learning curve of users: learning how to analyze vs. learning how to use the tool
*found unexpected use of the system - repurposing features
*identified usability issues
*design implications: tutorials
''Notes'':
*Add to to-read list, cite for Overview work
*among research questions, no explicit goal to better understand tasks
*listing 6 user case studies - but what are common traits? task/data abstractions?
*what was the methodology for studying users?
''Questions'':
*''MB'': re: user study - how much was studied during analysts' work, after analysts' work? Retrospective recall bias?
**both. considered user reports / stories / publications as objective data
*''Michael Sedlmair'': methods?
**Grounded theory, but not really
*''Niklas Elmqvist''
!!!Semantic Interaction for Sensemaking: Inferring Analytical Reasoning for Model Steering [TVCG]
!!!!<<cite Endert2012b>>
Alex Endert (PNNL), Patrick Fiaux, Chris North (VT)
*follow-up to ~FORCESpire CHI paper '12
*interactions impose a structure to the data that doesn't exist?
**no underlying DR in initial layout? where to start?
*a eval between manual card layout and layouts with semantic interaction
**system identified semantic synonyms of what users found to be important
**stages of insight based on entity weighting
**no users used the debug menu for model control aside from the training
*who were the users? what was the timeframe?
*conclusion: incremental formalism, implicit model steering, foraging to support sensemaking
''Questions'':
*''Melanie Tory'': implications for spatial memory?
**pinning helps to keep specific documents in place
*''Milton Paul'' (sp? govt analyst): documents in multiple piles? splitting/duplicating? can annotations really need to be pinned? read/unread indicators?
**items in multiple clusters can be pinned between clusters? annotations can be mined by something in the data, but some annotations can be free-flowing like "come back to this later", there's certainly space for both
*does combined semantic interaction techniques combine incrementally to accumulated insight
!!!Analystﲫspace: An Embodied Sensemaking Environment for Large, High Resolution Displays
!!!!<<cite Andrews2012>>
Christopher Andrews (Mt. Holyoke), Chris North (VT)
*large display workspace: "human-scale display" - focus on human limitations, not display possibilities
*distributed cognition primer
*why physical interaction rather than virtual interaction? the use of space: users make use of the entire large display workspace, whereas in virtual spaces, only a small proportion of available space is used.
*extraction of named entities, highlighting and notes; persisting as investigative waypoints, containment and co-occurrence links
*incremental formalism, creating structure as the analysis evolves - entirely manual (unlike ~FORCESpire) - includes align, distribute, timeline tools
''Notes'':
*Space to Think (earlier CHI paper)
''Questions'':
*''JS Yi'': same document classified in a multiple way? Is a single space sufficient?
**single space is sufficient, nor is space constrained to any one metaphor
!!!~SocialNetSense: Supporting Sensemaking of Social and Structural Features in Networks with Interactive Visualization
!!!!<<cite Gou2012>>
Liang Gou, Xiaolong (Luke) Zhang, Airong Luo, Patricia Anderson (Penn State, U. Michigan)
*social network analysis as a sensemaking process (Russell '93)
*current tools are function rich, but analytical provenance and process tracking is poor
*proposes s sensemaking framework for social network analysis
*NES: network exploring space, incorporates ~TreeNetViz (Gou & Zhang '11), representation space, process space
*tasks: zooming, grouping, selecting, annotating, nesting groups, expanding/collapsing nodes
''Notes'':
*cite for task taxonomy paper
*tool too complex for fast walkthrough, too much detail for talk
*too quiet
*why put up a url for 5s?
''Questions'':
*''Remco Chang'': interesting, but missing low-hanging fruit - switching back and forth b/w workspace and annotation space is disruptive: see Shrinivasan and van Wijk (CHI '08) - side by side more powerful?
**implications for visual complexity? use of floating panel?
*''Jeff Guenther'' (SFU): process management view?
**XML data format, tree structure format, recording state of representation
!!Space and Time + The Analysis Process
//Chair//: Bill Ribarsky + Brian Fisher
!!!A Visual Analytics Approach to Multi-scale Exploration of Environmental Time Series [TVCG]
!!!!<<cite Sips2012>>
Mike Sips, Patrick Kothur, Andrea Unger, ~Hans-Christian Hege, Doris Dransch (Potsdam)
*interval-based analysis of time-series plots, computing averages for fixed interval lengths, but what initial offset to select? what does this tell us?
*Pinus: hierarchy of scales and positions, matrix visualization of this hierarchy: derived data
*how to read the matrix view? start at the top and work down
*working with climate change scientists, they were able to detect trends in the interval-based Piunus matrix view that they couldn't have detected in the time-series line plot
''Notes'':
*Pinus?
*what is the task? spotting trends? anomalies?
*toy example
''Questions'':
*''Brian Fisher'': comparisons w/ wavelets? Fourier transform?
**automatic methods difficult to apply to this data due to uneven sampling rate / sampling window size. However the Pinus approach is intended to complement these other approaches
!!!Visual Analytics Methods for Categoric ~Spatio-Temporal Data
!!!!<<cite vonLandesberger2012>>
Tatiana von Landesberger, Sebastian Bremm, Natalia Andrienko, Gennady Andrienko, Maria Tekusova
*geotemporal trends w/ categories, clusters, tracking entity movement w/ linked views, global and local focus, overview and detail
*weather dataset, comparisons, similarities and clusterings
''Notes'':
*speedy slide changes
*objects change category across time points?
*what is the task?
*parallel coordinates / stacked bar charts with column dividing bars w/ varying levels of saturation leads to illusory 3D artifact (red foreground on blue background) what is the linked stacked histogram plot for? overview and detail?
*rainbow colour maps, highly saturated stacked bar chart
*what does saturation mean? occlusion? what do the colours mean? categories of weather? colour buckets?
*finished 5min early
''Questions'':
*''Silvia Miksch'': re: strict ordering of categories: if ordering changes, the patterns change
**if ordering is "nicer", it's easier to see the patterns, but changing the ordering doesn't hide the patterns. The ordering of categories is ~NP-hard.
!!!Watch This: A Taxonomy for Dynamic Data Visualization ᡦlt;<cite Cottam2012a>>
Joseph Cottam, Andrew Lumsdaine, Chris Weaver
*dynamic changes occurring throughout the visualization pipeline (e.g. streaming data)
*"if you're working at the level of items changing on the screen, it can sound a little funny" - [[Onion video about concentric circles|http://www.theonion.com/video/breaking-news-series-of-concentric-circles-emanati,14204/]]
*a taxonomy, a vocabulary for what is occurring on the screen with dynamic data
*quantifying/spatial categories (fixed, mutate, create, create/delete) and retinal categories (immutable, known scale, extreme bins, mutable scale) (Bertin): categories of dynamic change: identity preserving, transitional, immediate
**identify preserving transformations: fixed x [all], mutate x immutable
**transitional: mutate x [known scale, extreme bins], create x [immutable, known scale, extreme bins]
**immediate: mutate x mutable scale, create x mutable scale, create/delete x [all]
*type of transformation depends on task, decision tree for task
*examples: Gapminder, flow visualization, taxicab visualizations from NYC, London
*varying time scales, works best for streaming and dynamic queries, but not so well for reloading a new dataset, or interactivity
''Notes'':
*decision tree of tasks: cite/read for task work
*how was the taxonomy validated?
''Questions'':
*''Enrico Bertini'' (NYU Poly): (1) methodology followed to come up with model? (2) how do imagine people use your model?
*(1) we had a lot of videos, and categorized them, inter-coder reliability, agreement of categories, resolved ambiguities (2) it introduces a well-defined vocabulary, decision tree for tasks/process/questions for designers: guidelines
*''George Grinstein'': re: assumption that retinal and spatial variables are not independent. Taxonomy is not that clear cut. This is good first cut, but the two dimensions are so interrelated.
**We do recognize the ambiguities and redundancies that exist in the paper. "I'm going to hide behind Bertin who said that the two dimensions were separable enough to treat them that way."
!!!The User Puzzle - Explaining the Interaction with Visual Analytics Systems [TVCG] ᡦlt;<cite Pohl2012a>>
Margit Pohl, Michael Smuc, Eva Mayr
*theories to explain VA, problem solving, exploratory complex tasks where there is no clear solution/defined path: sensemaking, gestalt psychology, distributed cognition, graph presentation, skill-rule knowledge
**sensemaking: not guidelines, specific to tools, emphasis on what happens in analysts' mind
**gestalt theory: holistic, restructuring, insight, non-routine problems, but no design guidelines?
**distributed cognition: critical role of artifacts and interactions with artifacts, the design of the artifact, however the process of interaction is still not well understood
**graph comprehension: interacting with complex visualizations not well understood
**skill-rule knowledge: emphasis on errors, but hasn't been applied in VA / ~InfoVis, so we don't know if it can explain dynamic interaction
*comparing theories: schema for comparisons
**strengths and weaknesses: (labelled as high and low)
*take home: high-level theories don't explain TASKS, they don't give us design guidelines: we don't have cohesion, we need to extend theory, merge, and develop new theories.
''Notes'':
*"A warning that this presentation will contain almost no pictures", but this is still relevant for VA.
*begins with a definition of VA.
*high-level tasks: cite for task paper
*a meta-analysis of theories to explain VA
*Gestalt theory not well defined? no guidelines from Gestalt theory?!?!
*Distributed cognition: it's about interacting with artifacts, but we don't understand the interaction?
*no citations for latter 4 theories
*hard-to-read comparison plot the only visual in the slide deck
*I was hoping more out of the comparison
''Questions'':
*'''Bill Ribarsky'': re: interaction strategies, what about it? Which theory emphasizes reasoning through interaction?
**distributed cognition (DC) comes closest, but more is needed
*''Bill Ribarsky'': but does DC handle conversation b/w human and machine?
**it explains the communication with the data, perhaps not the tool?
*''Q'' what about Klein's data-frame sensemaking model, that does emphasize interaction: connect, refine, elaborate?
*''Enrico Bertini''? did you reorder columns and rows in the theory comparison plot to find patterns? (laughter from audience)
**No. ''Brian Fisher'': a ~VisWeek talk with one vis in and it gets criticized.
!!![Honorable Mention] Enterprise Data Analysis and Visualization: An Interview Study [TVCG]
!!!!<<cite Kandel2012>>
Sean Kandel, Andreas Paepcke, Joseph M. Hellerstein, Jeffrey Heer
*who are data analysts and what do they do?
*app users / scripters / hackers
*discover > wrangle > profile > model > report
*proposed research directions for VA research: what is/will be needed
*visualization needed more so for reporting, not for analysis
''Notes'':
*read in ~InfoVis reading group back in September
*good speaker, nice use of interviewee quotes, good summary of methods used
''Questions'':
*''Alex Andert'' (PNNL): the data-centric pipeline in a industrial domain, how does this match up with a cognitive pipeline, overlap with the intelligence domain? Future Work?
*''JS YI'': are there anti-visualization analysts?
**Yes. a few of those. "Visualizations get between me and my numbers". An issue of trust, they saw it a level of indirection. And also vice versa. Some people just wanted to see plots, didn't care to see numbers.
*''Mike'' (self-admitted scripter/hacker): re: jack-of-all-trades, master-of-none people. How should these people sell themselves on the job market?
**Lie, I guess? (laughter) Reading books? Learning new skills.
*''Q'': high-level models, non-domain specific tasks? (not really a question)
**(not understood)
!!TVCG on Visual Analytics
//Chair//: Silvia Miksch
!!!Evaluating the Role of Time in Investigative Analysis of Document Collections
!!!!<<cite Kwon2012>>
Bum chul Kwon, Waqas Javed, Sohaib Ghani, Niklas Elmqvist, Ji Soo Yi, David S. Ebert (Purdue)
*timelines in Jigsaw, Lifelines, ~GeoTime
*what is the role of temporal visualization in investigative analysis?
*controlled study: compared a tool w/ temporal visualization vs. a tool without for the same high-level task
*built a novel tool (~TimeInvestigator) (didn't use existing tool: easier to collect interaction data, easier to modify and control), supported brushing and linking b/w timeline view and document view (initial layout + manual regrouping and repositioning)
*methodology from Kang et al's 2009: controlled study approach, used same dataset and investigative task scenario. qualitative data collection: used Saraiya '06's longitudinal insight methodology: screen captures, insight reports, log files
*metrics: entities added/removed, alias detection, falsification, grouping entities
*interviews: adding temporal visualization allowed them to find entities faster
*guidelines: story building, lightweight qualitative evaluation: insights should be collected throughout the process, not just at end (retrospective recall bias), inter-coder reliability
''Notes'':
*a lot of unstated vocabulary specific to dataset - entities, aliases,
*controlled lab study (who were users?)
*comparative evaluation unfair
*tool not explained in detail: see the paper
*data collection and analysis adopted from different studies
*"report insights" button in the interface
*results unclear: what did they compare against? what was alternative condition? a reduced feature set?
*are they correctly using the term "falsification"?
*not all stories are linear in time
''Questions'':
*''Chris Andrews'' (Mt. Holyoke): benefit of temporal view hinges on the particular dataset used: couldn't frequency work just as well?
**many datasets have these strong temporal trends
!!Special Session: Visual Analytics in Practice
//Chair//: Jonathan C. Roberts
!!!Graphics, Infographics, and Data Visualization: An Economist's Call for Better Visuals
Jonathan A. Schwabish (JS) (Congressional Budget Office - CBO)
*economists, policy and budget analysts produce a lot of junk vis in their publications
*parade of bad graphics: e.g. Govt. Accountability Office's puzzle piece vis
*Familiarity with SAS, SPSS, more expertise needed in ~JavaSscript, D3, R
*congressional staffers are typically young, high turnover, high internal migration b/w departments: they need more of an overview, high-level understanding to get up to speed
*e.g. SNAP: CBO's supplemental nutrition program
*direction: from infographics to more interactivity: congressional staffers used to print out reports, now they print out infographics, soon they'll have them on tablets, then we can add interactivity
!!!Visualization in Industry
Kevin Lynagh (KL) (Kerning Labs, commercial visualization) @lynaghk
*cool stuff they've made (for money, solving real problems):
**wind energy tablet vis. real time wind speed and turbine data, no novel vis techniques, but a good business perspective
**bioinformatics / Harvard ~EdgeBio work (again, no novel vis, but domain problem solved)
**doc and patient tablet app for diabetes managements (novel vis. not required, often bar charts sufficient)
*tools: Clojure hipster programming language, D3, Chrome browser apps, open source work on Github
*cool stuff they're working on (research):
**bespoke visualization vs. exploratory visualization (e.g. bespoke suits vs. t-shirts) D3 vs. ~ggplot2
**while R is efficient, hard to find R programmers in industry, doesn't scale, vis needs to talk with Hadoop, Scala, ~JRuby
**re-writing ~ggplot2 in Clojure, de-complecting, associative data structures, recursion and faceting, (plot-as-geom paper by Grolemund and Wickham)
!!!Visualizations in an Ecosystem of Applications
Shahtab Wahid (SW) (Bloomberg UX, 731 Lexin, data analytics for stock traders)
*app store for thousands of financial visualizations, subscription service; most are single-screen, very focused purpose, a few core apps, some are multipurpose
*users: portfolio managers, analysts (researchers), traders (executing trades),
*workspace: multiple screens, Bloomberg apps, competitor apps, excel, outlook; screen real estate important, support passive and active interaction
*multi-purpose apps:
**BMAP - displaying middle-east oil fields, pipelines, container ships, exploratory, ongoing analysis
*single-purpose apps (dynamic, incoming data updates):
**quote lines (sparklines), tabular data, chart grids (line plots), OHLC/Candle charts (OHLC: open high low close), world trends, relative rotation graphs for historical data, IMAP (circular treemap divided by market sectors)
*two approaches:
**data -> multiple approaches / questions one might ask -> vis (multi-purpose)
**contextual inquiry, talking to customers, finding out what questions users want to ask -> generate vis alternatives -> single-purpose vis apps
*workflows with multi-purpose and single-purpose apps used in conjunction
**workflows are the priority, how to connect data across applications (how can brushing and linking occur across visualizations), large-scale ongoing sensemaking
!!!Discussion:
*''Roy R'' (Leeds): what was presented Visualization, not VA. What's happening at the analytics end? With analytical algorithms? Why things are the way they are. Is solution to just increase screen real estate, give more information? Why not give less information, but the right information, less screen space.
**''SW'': the user doesn't change the algorithms, flexibility may hurt efficiency. We can't affect the existing workflow, risk disrupting the market. More user study is required, maybe less information can make traders more efficient.
**''KL'': advanced computer algorithms not needed for many real-world problems.
*''JS Yi'': @JS, chartjunk in the healthcare industry. Why do we keep generating these problems? Where is the motivation for change?
**''JS'': Dierdre ~McClausky (Economist, U Chicago): more analysis is needed, integrated with communication. People don't think of graphics as a way to communicate the data, but is supplemental. So much data, shorter attention span: graphics are a way to communicate more quickly. Analysts receptive, but actually doing the work is not happening. To make good graphics, you don't need fancy programming languages or specific tools.
*''John Stasko'': @JS, to what degree are you partisan?
**''JS'': "we (CBO) are non-partisan. We do are job best when both sides hate us."
**''SW'': on bias, they have to be as unbiased as possible, because their newswire has market-shifting potential, e.g. stripping adjectives from news stories, revealing sources, transparency, objectivity.
*''Q'': @JS, whether economists take VA seriously as a means of analysis, or just Vis as a means of communication. Will tools like Tableau work for their datasets?
**attitude is that VA is the output for communication, not for analysis. A lot of the data is confidential / restricted. Use of tools like Tableau is growing
*''Q'': @SW, on workflow - do you change the workflow or build for the existing workflow? make the workflow more efficient? (2) individual / group users? designing for collaborative visualizations?
**(1) we don't have a tool for arranging workflows / tools. First-time users get to identify and tailor a workflow, setup. Highly individual. Identifying potential tools and setups, some will work for them, others not. (2) every person has their own setup, individually specified; in team situations, little control over which tools/workflows are used
*''Q'': do analysts need to defend their workflow? Should we prescribe workflows?
**''SW'': In-house reports to defend trading choices would not be specific enough to individual applications or workflows
**''JS'': 230 CBO analysts, each doing their own workflows (more standardization?)
*''Q'': Why is there so much low-hanging fruit? Excel? Why do we keep reinventing the wheel? The problem seems central to the most-used tools
**''JS'': R Kosara says: of 18 Excel chart types, you should only use 4
!~SciVis Sessions
!!Evaluation
//Chair//: Dan Keefe
!!!Human Computation in Visualization: Using Purpose Driven Games for Robust Evaluation of Visualization Algorithms
!!!!<<cite Ahmed2012>>
Nafees Ahmed, Ziyi Zheng, Klaus Mueller (Stony Brook)
*Gamification in visualization; crowdsourcing
*Evaluating 4 algorithms for colour blending
*[[Disguise|http://vail.cewit.stonybrook.edu/Projects/HPU/Disguise]], a arcade-style survival game, involves clicking on occluding shapes, 塬th rewards/penalties based on whether occluded/occluder is clicked, incredibly rapid
**261 players, 21 million data points, 14.6 inputs per minute
**confusion evident between occluded/occluding foreground/background, transparency
*Implications for gamifying an experiment:
**it's free! players are engaged, trying to optimize their score, not monetary reward
**long time required to develop game
''Notes'':
*Marketing buzzword bingo!
''Questions'':
*''Q'': What about situations where you don't know ground truth?
**case-by-case basis?
*''Q'': How long to develop game? How much can you scale? How sophisticated of a visualization research question can you ask via a game? The concept of the game is simple; how engaged could users be?
**approx. 1 month full-time coding. Users seemed to be engaged, also was running out of time.
!!!Evaluation of Multivariate Visualization (MVV) on a Multivariate Task
!!!!<<cite Livingston2012>>
Mark A. Livingston, Jonathan W. Decker, Zhuming Ai (US Naval Research Lab)
*How much data can you show? What is the task? How much data can you show before it changes the task?
*MVV: oriented slivers (Weigle '00), data-driven spots (Bokinsky '03), attribute blocks (heatmap, Miller '07)
*survey of comparisons of previous MVV comparison studies; includes list of tasks (cite/flag?)
*Decided upon new task: finding maximum overlap of multivariate values (do users actually do this? example domains?)
*Experimental method comparing ~MVVs: error, answers, time, workload
*findings: strong effect of MVV type, interaction effects w/ # layers and time, error, interactions with target size and time, error
*positive correlation b/w target density and error, as targets became denser, user error increased: counterintuitive? Density is distracting?
''Notes'':
*too much text on slides, too fast on experimental design, hypotheses
*what is target domain? who does the task? where were users recruited from?
*errors 㯲rect answers? what is "answers" metric?
''Questions'':
*''Q'': Tasks? How to decide?
**inspired by cartographic data; inspired by tasks used/reported in literature, tasks are general low-level analysis task that hopefully generalize
!Panel: Reproducible Visualization Research: How Do We Get There?
//Chair//: Enrico Bertini
!!Grassroots reproducibility: Just do it!
Tamara Munzner (TM) (UBC)
!!Fairly sharing the costs of reproducibility
Gordon Kindlmann (GK) (Utah)
*Reproducibility as sharing costs
*computational results: ~WaveLab quote - the paper is advertisement for the computational results
*critique when not comparing against previous work, but previous work is not accessible: cost of reimplementing previous work
*cost of hosting archived supplemental material
*cost associated with being scooped on your own future work
*no single solution, no mandate for reproducibility
*cost sharing: author, author's advisor, institution, support staff, research (sub)communities, reviewers, ~VisWeek, IEEE TVCG, publishers of supplemental materials + updates
*Resources: [[EuroVis panel on reproducibility|eurovvv.org]]
!!Reproducibility: It's good for you!
Juliana Freire (NYU Poly) (presented by Carlos Scheidegger (CS), AT&T Research)
*[[slides|http://bit.ly/revis2012]] available w/ links
*essential for science: ETH, NSF guidelines
*we can't solve a cultural problem with a technological solution
*a different standard than natural science
*~VisTrails - papers with reproducible / executable figure
*slides created with deck.js
!!Reproducible Research?
Tim Dwyer (TD) (Microsoft)
*the Microsoft black box patent cube award
*the patent system is imperfect, but corporations still like them; but publishing papers is good PR, helps community; however source code will not be shared
*story about ~VisualStudio node-link and matrix view visualizations for static analysis of code dependencies; MS made source code available, but got beaten to market by a rival
*should researchers and authors be software publishers? papers should be the primary artifact, not the code/results
!!Discussion
*''Q'': Gordon Donohue (Stanford Mathematician) quote on why he was so highly cited: "implement the method in software, describe the software in the paper, give software away for free", "distribute over the internet"
*''Lyn Bartram'': anonymzying data in design studies is hard - an ethics boards issue; copyright issues.
**''TM'': I have an ethics approval in place that will let me release user study data. Security through obscurity. But danger of messing up ethics. Lab and field study differences.
*''Heidi Lam'': on qualitative data w/ software logging; knowing that you're being logged changes behaviour; knowing that it will be public also changes behaviour. On quant. data: I have terabytes. Even if I wanted to share, how? e.g. AOL user data being released, identities being revealed, researchers fired. Netflix user data pulled from Netflix challenge.
**''CS'': transparency is key. It makes our lives harder, but what's the alternative. Re: quant data: NSF has guidelines for storing data, maintaining, hosting it. The alternative is much worse.
*''Enrico Bertini'': ethics approval process can overlook a lot
**''GK'': a lot of these issues are human issues. Reproducibility is about more, about sharing code, data, parameters - non-human issues
**''TM'': "Every time a CS dept. sysadmin breaks a link to an archived research website mentioned in a paper, God kills a kitten. It's a sin."
**''TD'': papers should not omit critical material: "Imagine if Newton's Principia contained critical links to ~URLs that got pulled in 1722"
*''Paul Rosenthal'': What is the timeline for encouraging/rewarding reproducible research?
**''TM'': it won't happen overnight. No mandates. What could we mandate anyway?
**''CS'': we need to do the extra work. we need critical mass.
**''TD'': what can IEEE do for me to provide reproducibility outside the paywall?
**''GKL'': recognize, not require reproducibility
*''Patrick Butler'' (VT): tools other than ~VisTrails? Graph reproduction on ~MatPlotLib, exact parameters change, systems change.
**''CS'': make :), shell scripts, github (used for offsite backup, also the cool thing to do - ~OctoCat loves you)
*''Edgar C. Olson'' (UW): GMU provides hosting of modelling software, keeps code and data private, but users could run software without revealing the source. Mercurial (sp?) version tracking as alternative to github
**''GK'': best practices can arise from these grassroots initiatives
*''Michael Wyebrow'' (sp?) (Monash U): universities and funding bodies need to recognize this extra work. technical reports? source code as research outcomes? what is the academic currency?
**''GK'': Absolutely. Shared costs and currency. ITK journal in image processing has strict reproducibility guidelines.
**''TM'': UBC starting to recognize copyrights for software releases for tenure/promotion.
**''CS'': recognition has to occur for paper authors and software authors. "I've stared at the enemy and the enemy is us".
*''Daniel Keim'' (Konstanz): word of caution. Reproducible at 10+ years can't happen. Even at 2-3 years, libraries, drivers change - unless you have a user base. Opposing the requirement of having it run.
**''TM'': difference b/w stuff I released 20 years ago and stuff I didn't - the former was maintained, got rolled into ~FreeBSD release (but haven't been able to recompile herself in 7+ years) the latter bitrotted. No mandates: no one here is arguing that.
*''Enrico Bertini'': our discussion has veered toward reproducibility = open source, which is limiting. Reproducibility can take other forms, and appear in the paper: proofs, pseudocode, images, parameters.
**''CS'': paper limits are not necessary, 10 page limits are for reviewers. Everything after 10 pages can be included but not reviewed.
*''Q'': //Continuous Delivery//, staying up to date on software practices, continuous integration
**''GK'': we cannot expect everyone to stay up-to-date on these practices, domains are so disjoint.
**''Srinivasan'' (KAUST): (1) mixed messages on open source vs. being unaware of software integration practices (2) borrowing from software testing.
**''TD'': software testing is often the first result to go stale.
**''CS'': github is just one of the venues for this.
*''Q'': CHI videos from 80s recently got pulled from ACM/DL. All these novel interaction demonstrations lost. Who should maintain? Who is responsible?
**''CS'': "this is equivalent to destroying a sketch by Picasso".
**''TM'': these things have value; e.g. the role of emulators, retro video games becoming cool again
*''Tim Lebo'' (RPI): aside from the source code, what about the data?
**''GK'': it's a blessing to publish collaborator's data. If not, publish synthetic data. Data is capital.
**''TM'': data sanitization may need to occur, even at 11th hour, collaborators don't want to get scooped in their own domain
**''CS'': publish both real and synthetic.
*''Mark Livingston'': the role of contest datasets, benchmarks, publicly available data. Researchers can add to these.
*''Q'': difficulties with biological datasets, provenance, labs / research centres unwilling to give up data. //Workflow Forever// initiative, Software Sustainability Institute in the UK
!Panel: Quality of Visualization: The Bake Off
//Chair//: Min Chen (MC)
>//What does 쩴y�when discussing visualization? Algorithmic quality? Empirical studies? Measurements of quality? Real-world users? All or none of the above?//
Based on Quality of Service (~QoS) in telephony (38K publications). Quality of service for Visualization (~QoV) (5 publications). What is the quality of visualization? How should we establish the protocols for assessing the quality of visualization? The "Transfer Function Bake Off".
!!School A for "Algorithms or Automation"
M. Eduard Gr⠨MEG) (TU Vienna)
*al-Chwarizmi (780-835): the term algorithm derived from his name
*quality of algorithms
*we have algorithms with so many parameters to tune, too many for many domain users
*intrinsic evaluation of algorithms: value of sensitivity analysis to explore algorithm/parameter space tuning
*extrinsic evaluation of algorithms: different data, benchmarks,
**e.g. ~AlgoLets: multiple data streams, multiple algorithms integrating multiple data portions and producing image fragments (important in VA, ~InfoVis)
*the future: integrated views (Balabanian et al '10), fuzzy visualizations, comparative visualizations (Malik et al '10)
!!School E for "Experiments or Empirical Studies"
Penny Rheingans (PR) (U Maryland Baltimore County)
*"tree falls in the forest" - perception is needed: if a visualization displays in an empty room, does it have values?
*quantitative user studies, qualitative evaluation
*counting the number of discoveries results in uneven results: did a discovery occur? what is the quality of that discovery?
*focusing on experimental lab studies
**a primer on lab studies: determining the conceptual variable: what can we measure? what's a task? what's a task that's simple enough to train someone quickly but complex enough to tell us something interesting. balancing expertise and availability of participants. how can I automate the administration of the experiment?
**how to review? what is the conceptual variable? does it matter? is the comparison/control case fair and meaningful? discussion of results, anomalies, generalities; what was expected/unexpected? is it replicable? construct and internal validity.
!!School M for Metrics or Measurement
Matthew O. Ward (MOW) (Worcester Poly. Institute)
*"I couldn't disagree with you [Penny] more"
*we can use metrics to search a space of visualizations such that they convey the optimal amount of work
*currently used measures are simple, limited
*what do we really want to measure?
**information theory: visualization as a communication model w/ minimal distortion
**measure visual transformation artifacts, Tufte's vis lies, the effectiveness asymptote
*measuring information content: counting relations b/w records, attributes - but a lot of redundancy
*measuring information loss w/ sampling, clustering, aggregation, occlusion, rotation (e.g. 3D geospatial population visualization)
**example: measuring information loss in sampled parallel coordinates
*the user is not involved in these metrics
*easier to measure information loss than information gained, but both approaches have their validity. provide these measures to the users so they have an accurate understanding of what they're seeing, acknowledging what information was lost
!!School R for "Real Users and Real World Applications"
Kelly Gaither (KG) (U. Texas, Austin)
*"with all respect to my colleagues, it's time to get real."
*//"the purpose of computing is insight, not numbers"// - RW Hamming (1961)
*the purpose of visualization is insight, not images. We don't need to create something new if an existing tool/technique works. We're leveraging a capability of our visual system. There is no substitute for sitting down with users and understanding objectives, learn about their work, building a trust relationship
*establish a set of guiding questions: use them to guide, they will change, but they give us a starting point
*don't reinvent the wheel: there's lots of questions out there
*emphasis on the role of collaboration: collaboration is magic
!!Discussion
*''Steve Haroz'': none of the panelists disagreed with one another.
**'''MEG'': it depends on the goals. e.g. algorithm optimization - no need for users.
**''PR'': I'm also a fan of the real world. But the problem is "how does it transfer to somewhere else?" We don't understand magic. We need to explain what we find in the real world, so we measure in the lab.
**''KG'': we all agree with one another. But these evaluations can be prioritized given a question.
*''Stefan'': what is the current state of to what extent / state user studies and screen measures are effective? (question unclear)
**''PR'': we're trying to ask: what does it mean to be effective? When we are inundated with qualitative user response data, how to make sense of it. The gap between that and quantitative data is still evident.
*''V. Lemieux'' (UBC): on insights and role of interaction in visual analytics. (ramble question)
**''MOW'': transformation insight
*''Q'': on using multiple datasets?
**''MEG'': in CS, if your algorithm works on 3 datasets, it works on all datasets. This is perhaps why our tools aren't widely adopted in the real world. Datasets are often subtly/minutely different. The community should reward evaluation based on lots of techniques and datasets.
**''MOW'': in ML community, you need to evaluate your technique w/ multiple datasets. Our problem is that we have to include figures. Perhaps we should encourage this to be included in supplemental material.
**''PR'': we often work with collaborators, with //THE// dataset. Anything that creates distance b/w collaborators is a double-edged sword. In ML, they have standardized repositories of datasets, they don't need to describe the datasets in their papers.
**''KG'': we may need a better (more?) repository of datasets, better (more?) open-source packages
*''Q'' (Air Force Research Lab): "Until we can smell information better, we need to use our eyes". Are we a self-licking ice cream cone? Should we be defining the quality ourselves? What is the halting problem of visualization? Should we be in domain conferences? Should we be in domain conferences? Is visualization a science or are domain sciences the science? We'll be done when all contributions are domain solutions. When we no longer offer generalizations, we'll be at domain conferences. Unless in 20 years we'll be at ~SmellWeek.
**''KG'' we'll always have a problem. Data is always changing. Size, type, provenance, heterogeneity, variable quality. We're still a science. We'll see you in 20 years
**''MEG'': I don't see visualization dying out as a science. Maybe there will be a time like text processing (no more conferences in that field). We're not yet done.
**''PR'': the data will evolve, and we're dealing with human learning, and human understanding. It's a deep problem, and it's something that's not going to be solved soon. We'll still have jobs.
**''MOW'': we'll still have jobs.
*''Torsten M⧧: Quality of Service (~QoS) definition and the role of guidelines? Confusion over Quality and Evaluation. Where is the threshold for high quality? What about
**''PR'': we can't teach everyone to do the right thing, general guidelines are hard because problems are so context-dependent. Defining quality has to be done on a per-basis.
**''KG'': it's about how well we answer the questions, not about aesthetics or the frequency of "a-ha" moments.
*''Heidi Lam'': How can we measure quality in user studies?
*''Q'': how can we separate the quality of the users from the quality of the visualization? the quality of domain scientist collaborators?
**''PR'': there will always be superior users and collaborators, regardless, inferior collaborators still need vis
!!!Wrap up:
*''KG'': quality needs to be assessed throughout the whole process, start to finish.
*''MOW'': agrees with ''KG''. We need to know more about information theory metrics.
*''PR'': the presence of the user is key, despite what ''MEG'' and ''MOW'' say. The user is implicit in their work too. And in ''KG'''s work, the users have names and stories and jobs, but sometimes users are anonymous and must generalize
*''MEG'': quality of algorithms ᬩty of visualization. Algorithm users need to understand the context of the algorithm (users included) more.
*''MC'': on quality standpoints and the -isms in psychology. ''MOW'' is a structuralist, ''MEG'' is a functionalist, ''KG'' is a pragmatist, ''PR'' is a behaviourist. visualization is a science: visology?
!Misc Notes / Links
*[[D3 tutorial|http://www.jeromecukier.net/blog/2012/10/15/d3-tutorial-at-visweek-2012/]]
*[[Adaptive Composite Map Projections|http://vimeo.com/47482303]] - A new composite map projection for web maps that adapts to map scale and the shown geographic area, developed by the Cartography and Geovisualization Group at Oregon State University.
!References
<<bibliography VisWeek2012 showAll>>
{{{
@inproceedings{Kang2011,
author = {Kang, Y. A. and Stasko, J. T.},
title = {{Characterizing the Intelligence Analysis Process: Informing Visual Analytics Design through a Longitudinal Field Study}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {19--28},
volume = {17},
year = {2011}
}
@article{Hullman2011a,
author = {Hullman, J. and Adar, E. and Shah, P.},
doi = {10.1109/TVCG.2011.175},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2213--2222},
title = {{Benefitting InfoVis with Visual Difficulties.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034340},
volume = {17},
year = {2011}
}
@article{Bae2011,
author = {Bae, J. and Watson, B.},
doi = {10.1109/TVCG.2011.187},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
month = dec,
number = {12},
pages = {2268--75},
title = {{Developing and evaluating quilts for the depiction of large layered graphs.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034346},
volume = {17},
year = {2011}
}
@article{Brandes2011,
author = {Brandes, U. and Nick, B.},
doi = {10.1109/TVCG.2011.169},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2283--90},
title = {{Asymmetric relations in longitudinal social networks.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034348},
volume = {17},
year = {2011}
}
@article{Albuquerque2011a,
author = {Albuquerque, G. and Lowe, T. and Magnor, M.},
doi = {10.1109/TVCG.2011.237},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2317--24},
title = {{Synthetic generation of high-dimensional datasets.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034352},
volume = {17},
year = {2011}
}
@article{Albers2011,
author = {Albers, D. and Dewey, C. and Gleicher, M.},
doi = {10.1109/TVCG.2011.232},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2392--401},
title = {{Sequence surveyor: leveraging overview for scalable genomic alignment visualization.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034360},
volume = {17},
year = {2011}
}
@article{Steinberger2011,
author = {Steinberger, M. and Waldner, M. and Streit, M. and Lex, A. and Schmalstieg, D.},
doi = {10.1109/TVCG.2011.183},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2249--58},
title = {{Context-preserving visual links.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034344},
volume = {17},
year = {2011}
}
@inproceedings{Thorson2011,
author = {Thorson, L. and Downing, J. and Ellingson, A. and Nuckley, D. and Keefe, D. F.},
title = {{A Designerᰰroach to Scientific Visualization: Visual Strategies for Illustrating Motion Datasets}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
year = {2011}
}
@inproceedings{Anand2011,
author = {Anand, A. and Wilkinson, L. and Dang, T. N.},
title = {{Using Random Projections to Identify Class-Separating Variables in High-Dimensional Spaces}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {261--262},
year = {2011}
}
@article{Cao2011,
author = {Cao, N. and Gotz, D. and Sun, J. and Qu, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2581--2590},
title = {{DICON: Interactive Visual Analysis of Multidimensional Clusters}},
volume = {17},
year = {2011}
}
@article{Borkin2011,
author = {Borkin, M. A. and Gajos, K. Z. and Peters, A. and Mitsouras, D. and Melchionna, S. and Rybicki, F. J. and Feldman, C. L. and Pfister, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2479--2488},
title = {{Evaluation of Artery Visualizations for Heart Disease Diagnosis}},
volume = {17},
year = {2011}
}
@article{Turkay2011,
author = {Turkay, C. and Filzmoser, P. and Hauser, H.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2591--2599},
title = {{Brushing Dimensions 䵡l Visual Analysis Model for High-Dimensional Data}},
volume = {17},
year = {2011}
}
@article{Wickham2011,
author = {Wickham, H. and Hofmann, H.},
doi = {10.1109/TVCG.2011.227},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2223--30},
title = {{Product plots.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034341},
volume = {17},
year = {2011}
}
@inproceedings{Wang2011,
author = {Wang, X. and Dou, W. and Butkiewicz, T and Bier, E. A. and Ribarsky, W.},
title = {{A Two-Stage Framework for Designing Visual Analytics System in Organizational Environments}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {249--258},
year = {2011}
}
@inproceedings{Heer2011,
author = {Heer, J. and Perer, A.},
title = {{Orion: A System for Modeling, Transformation and Visualization of Multidimensional Heterogeneous Networks}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
keywords = {data management,data trans-,end-user programming,formation,graphs,social network analysis,visualization},
pages = {49--58},
year = {2011}
}
@inproceedings{Smith2011,
author = {Smith, A. S. and Xu, W. and Sun, Y. and Faeder, J. R. and Marai, G. E.},
title = {{RuleBender: Integrated Visualization for Biochemical Rule-Based Modeling}},
booktitle = {Proc. IEEE Symp. Biological Data Visualization (BioVis)},
pages = {103--110},
year = {2011}
}
@article{Bostock2011,
author = {Bostock, M. and Ogievetsky, V. and Heer, J.},
doi = {10.1109/TVCG.2011.185},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2301--9},
title = {{Dᴡ-driven documents.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034350},
volume = {17},
year = {2011}
}
@article{Isenberg2011a,
author = {Isenberg, P. and Bezerianos, A. and Dragicevic, P. and Fekete, J. D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2469--2478},
title = {{A Study on Dual-Scale Data Charts}},
volume = {17},
year = {2011}
}
@article{Bertini2011,
author = {Bertini, E. and Tatu, A. and Keim, D.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2203--2212},
title = {{Quality Metrics in High-Dimensional Data Visualization: An Overview and Systematization}},
volume = {17},
year = {2011}
}
@article{Walny2011,
author = {Walny, J. and Carpendale, S. and Riche, N. H. and Venolia, G. and Fawcett, P.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2508--2517},
title = {{Visual Thinking In Action: Visualizations As Used On Whiteboards}},
volume = {17},
year = {2011}
}
@inproceedings{Albuquerque2011,
author = {Albuquerque, G. and Eisemann, M. and Magnor, M.},
title = {{Perception-Based Visual Quality Measures}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {11--18},
volume = {17},
year = {2011}
}
@article{Wood2011,
author = {Wood, J. and Badawood, D. and Dykes, J. and Slingsby, A.},
doi = {10.1109/TVCG.2011.174},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2384--91},
title = {{BallotMaps: Detecting Name Bias in Alphabetically Ordered Ballot Papers.}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034359},
volume = {17},
year = {2011}
}
@article{Rodgers2011,
author = {Rodgers, J. and Bartram, L.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2489--2497},
title = {{Exploring Ambient and Artistic Visualization for Residential Energy Use Feedback}},
volume = {17},
year = {2011}
}
@inproceedings{Patro2011,
author = {Patro, R. and Cho, S. S. and Thirumalai, D. and Varshney, A.},
title = {{MDMap: A System for Data-Driven Layout and Exploration of Molecular Dynamics Simulations}},
booktitle = {Proc. IEEE Symp. Biological Data Visualization (BioVis)},
pages = {111--118},
year = {2011}
}
@article{Lloyd2011,
author = {Lloyd, D. and Dykes, J.},
title = {{Human-centered approaches in geovisualization design: investigating multiple methods through a long-term case study.}},
doi = {10.1109/TVCG.2011.209},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2498--2507},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22034371},
volume = {17},
year = {2011}
}
@inproceedings{Fisher2011,
author = {Fisher, D.},
title = {{Incremental, Approximate Database Queries and Uncertainty for Exploratory Visualization}},
booktitle = {Proc. IEEE Symp. Large-Scale Data Analysis and Visualization (LDAV)},
pages = {73--80},
year = {2011}
}
@inproceedings{Kwon2011,
author = {Kwon, B. C. and Fisher, B. and Yi, J. S.},
title = {{Visual Analytic Roadblocks for Novice Investigators}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {1--9},
volume = {17},
year = {2011}
}
@inproceedings{Ziemkiewicz2011,
author = {Ziemkiewicz, C. and Crouser, R. J. and Yauilla, A. R. and Su, S. L. and Ribarsky, W.},
title = {{How Locus of Control Influences Compatibility with Visualization Style}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {79--88},
volume = {17},
year = {2011}
}
}}}
{{{
@inproceedings{Slingsby2012,
author = {Slingsby, A. and Dykes, J.},
title = {{Experiences in involving analysts in visualisation design}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Silva2012,
author = {Silva, I. C. S. and Freitas, C. M. D. S. and Sasso, D. and Santucci, G.},
title = {{An integrated approach for evaluating the visualization of intensional and extensional levels of ontologies}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Cushing2012,
author = {Cushing, J. B. and Hayduk, E. and Walley, J. and Winters, K. and Lach, D. and Stafford, S.},
title = {{Which visualizations work, for what purpose, for whom? Evaluating visualizations of terrestrial and aquatic systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Jackson2012,
author = {Jackson, B. and Coffey, D. and Thorson, L. and Schroeder, D. and Ellingson, A. M. and Nuckley, D. J. and Keefe, D. F.},
title = {{Toward mixed method evaluations of scientific visualizations and design process as an evaluation tool}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Anderson2012,
author = {Anderson, E. W.},
title = {{Evaluating scientific visualization using cognitive measures categories and subject descriptors}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Peck2012,
author = {Peck, E. M. and Yuksel, B. F. and Harrison, L. and Ottley, A. and Chang, R.},
title = {{ICD 3: Towards a 3-dimensional model of individual cognitive differences}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Endert2012,
author = {Endert, A. and North, C.},
title = {{Interaction junk: User interaction-based evaluation of visual analytic systems}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Cottam2012,
author = {Cottam, J. A. and Lumsdaine, A.},
title = {{Spatial autocorrelation-based information visualization evaluation}},
volume = {0},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Dasgupta2012,
author = {Dasgupta, A. and Kosara, R.},
title = {{The importance of tracing data through the visualization pipeline}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Gleicher2012,
author = {Gleicher, M.},
title = {{Stop the evaluation arms race! A call to evaluate visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Meyer2012,
author = {Meyer, M. and Sedlmair, M. and Munzner, T.},
title = {{The four-level nested model revisited: blocks and guidelines}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Elmqvist2012,
author = {Elmqvist, N. and Yi, J. S.},
title = {{Patterns for visualization evaluation}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Scholtz2012,
author = {Scholtz, J. and Whiting, M. A. and Plaisant, C. and Grinstein, G.},
title = {{A reflection on seven years of the VAST challenge}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Mcnamara2012,
author = {McNamara, L. A. and Orlando-Gay, N.},
title = {{Reading, sorting, marking, shuffling: Mental model formation through information foraging}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Kaastra2012,
author = {Kaastra, L. T. and Arias-Hernandez, R. and Fisher, B.},
title = {{Evaluating analytic performance}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Kim2012,
author = {Kim, S. H. and Yun, H. and Yi, J. S.},
title = {{How to filter out random clickers in a crowdsourcing-based study?}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Forsell2012,
author = {Forsell, C. and Cooper, M.},
title = {{Questionnaires for evaluation in information visualization}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Pohl2012,
author = {Pohl, M.},
title = {{Methodologies for the analysis of usage patterns in information visualization}},
booktitle = {Proc. WS. BEyond time and errors: novel evaLuation methods for Information Visualization (BELIV)},
year = {2012}
}
@inproceedings{Anand2012,
author = {Anand, A. and Wilkinson, L. and Dang, T. N.},
title = {{Visual pattern discovery using random projections}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {43--52},
year = {2012}
}
@article{Bezerianos2012,
author = {Bezerianos, A. and Isenberg, P.},
doi = {10.1109/TVCG.2012.251},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2516--2525},
title = {{Perception of visual variables on tiled wall-sized displays for information visualization applications}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327257},
volume = {18},
year = {2012}
}
@inproceedings{Maguire2012,
author = {Maguire, E. and Rocca-Serra, P. and Sansone, S. A. and Davies, J. and Chen, M.},
title = {{Taxonomy-based glyph design 䨠a case study on visualizing workflows of biological experiments}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {12},
pages = {2603--2612},
volume = {18},
year = {2012}
}
@article{Talbot2012,
author = {Talbot, J. and Gerth, J. and Hanrahan, P.},
doi = {10.1109/TVCG.2012.196},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2613--2620},
title = {{An empirical model of slope ratio comparisons}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327267},
volume = {18},
year = {2012}
}
@article{Kong2012a,
author = {Kong, N. and Agrawala, M.},
doi = {10.1109/TVCG.2012.229},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2631--2638},
title = {{Graphical overlays: Using layered elements to aid chart reading}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327269},
volume = {18},
year = {2012}
}
@article{Lee2012a,
author = {Lee, B. and Isenberg, P. and Riche, N. H. and Carpendale, S.},
doi = {10.1109/TVCG.2012.204},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2689--2698},
title = {{Beyond mouse and keyboard: Expanding design considerations for information visualization interactions}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327275},
volume = {18},
year = {2012}
}
@article{Tominski2012,
author = {Tominski, C. and Schumann, H. and Andrienko, G. and Andrienko, N.},
doi = {10.1109/TVCG.2012.265},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2565--2574},
title = {{Stacking-based visualization of trajectory attribute data}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327262},
volume = {18},
year = {2012}
}
@inproceedings{Zhang2012,
author = {Zhang, L. and Mittelst, S. and Weber, S.},
title = {{Visual analytics for the big data era: A comparative review of state-of-the-art commercial systems}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {173--182},
year = {2012}
}
@inproceedings{Piringer2012,
author = {Piringer, H. and Buchetics, M. and Benedik, R.},
title = {{AlVis: Situation awareness in the surveillance of road tunnels}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {153--162},
year = {2012}
}
@article{Lam2011,
author = {Lam, H. and Bertini, E. and Isenberg, P. and Plaisant, C. and Carpendale, S.},
doi = {10.1109/TVCG.2011.279},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
pages = {1--18},
title = {{Empirical studies in information visualization: Seven scenarios}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22144529},
year = {2011}
}
@article{Block2012,
author = {Block, F. and Horn, M. S. and Phillips, B. C. and Diamond, J. and Evans, E. M. and Shen, C.},
doi = {10.1109/TVCG.2012.272},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2789--2798},
title = {{The DeepTree exhibit: Visualizing the tree of life to facilitate informal learning}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327285},
volume = {18},
year = {2012}
}
@article{Borgo2012,
author = {Borgo, R. and Abdul-Rahman, A. and Mohamed, F. and Grant, P. W. and Reppa, I. and Floridi, L. and Chen, M.},
doi = {10.1109/TVCG.2012.197},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2759--2768},
title = {{An empirical study on using visual embellishments in visualization}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327282},
volume = {18},
year = {2012}
}
@article{Boukhelifa2012,
author = {Boukhelifa, N. and Bezerianos, A. and Isenberg, T. and Fekete, J. D.},
doi = {10.1109/TVCG.2012.220},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2769--2778},
title = {{Evaluating sketchiness as a visual variable for the depiction of qualitative uncertainty}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327283},
volume = {18},
year = {2012}
}
@inproceedings{Butler2012,
author = {Butler, P. and Chakraborty, P. and Ramakrishnan, N.},
title = {{The Deshredder: A visual analytic approach to reconstructing shredded documents}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {113--122},
year = {2012}
}
@article{Crouser2012,
author = {Crouser, R. J. and Chang, R.},
doi = {10.1109/TVCG.2012.195},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2859--2868},
title = {{An affordance-based framework for human computation and human-computer collaboration}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327292},
volume = {18},
year = {2012}
}
@article{Gomez2012b,
author = {Gomez, S. R. and Jianu, R. and Ziemkiewicz, C. and Guo, H. and Laidlaw, D.},
doi = {10.1109/TVCG.2012.214},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2411--2420},
title = {{Different strokes for different folks: Visual presentation design between disciplines}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327246},
volume = {18},
year = {2012}
}
@article{Haroz2012a,
author = {Haroz, S. and Whitney, D.},
doi = {10.1109/TVCG.2012.233},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2402--2410},
title = {{How capacity limits of attention influence information visualization effectiveness}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327245},
volume = {18},
year = {2012}
}
@article{Heimerl2012,
author = {Heimerl, F. and Koch, S. and Bosch, H. and Ertl, T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2839--2848},
title = {{Visual classifier tTraining for text document retrieval}},
volume = {18},
year = {2012}
}
@article{Hofmann2012a,
author = {Hofmann, H. and Follett, L. and Majumder, M. and Cook, D.},
doi = {10.1109/TVCG.2012.230},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2441--2448},
title = {{Graphical tests for power comparison of competing designs}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327249},
volume = {18},
year = {2012}
}
@inproceedings{Kandogan2012,
author = {Kandogan, E.},
title = {{Just-in-time annotation of clusters, outliers , and trends in point-based data visualizations}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {73--82},
year = {2012}
}
@article{Kim2012a,
author = {Kim, S. H. and Dong, Z. and Xian, H. and Upatising, B. and Yi, J. S.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2421--2430},
title = {{Does an eye tracker tell the truth about visualizations? Findings while wnvestigating visualizations for decision making}},
volume = {18},
year = {2012}
}
@article{Ma2012,
author = {Ma, J. and Liao, I. and Ma, K. L. and Frazier, J.},
doi = {10.1109/TVCG.2012.244},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2799--2808},
title = {{Living liquid: Design and evaluation of an exploratory visualization tool for museum visitors}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327286},
volume = {18},
year = {2012}
}
@article{MacEachren2012,
author = {MacEachren, A. M. and Roth, R. E. and O'Brien, J. and Li, N. and Swingley, D. and Gahegan, M.},
doi = {10.1109/TVCG.2012.279},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2496--2505},
title = {{Visual semiotics \& uncertainty visualization: An empirical study}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327255},
volume = {18},
year = {2012}
}
@article{Marriott2012,
author = {Marriott, K. and Purchase, H. and Wybrow, M. and Goncu, C.},
doi = {10.1109/TVCG.2012.245},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2477--2485},
title = {{Memorability of visual features in network diagrams}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327253},
volume = {18},
year = {2012}
}
@article{Micallef2012,
author = {Micallef, L. and Dragicevic, P. and Fekete, J. D.},
doi = {10.1109/TVCG.2012.199},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2536--2545},
title = {{Assessing the effect of visualizations on Bayesian reasoning through crowdsourcing}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327259},
volume = {18},
year = {2012}
}
@article{Pileggi2012,
author = {Pileggi, H. and Stolper, C. D. and Boyle, J. M. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2819--2828},
title = {{SnapShot: Visualization to propel ice hockey analytics}},
volume = {18},
year = {2012}
}
@article{Pilhofer2012,
author = {Pilh적. and Gribov, A. and Unwin, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2506--2515},
title = {{Comparing clusterings using Bertin㠩dea}},
volume = {18},
year = {2012}
}
@article{Sedlmair2012c,
author = {Sedlmair, M. and Meyer, M. and Munzner, T.},
doi = {10.1109/TVCG.2012.213},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2431--2440},
title = {{Design study methodology: Reflections from the trenches and the stacks}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327248},
volume = {18},
year = {2012}
}
@inproceedings{Tatu2012,
author = {Tatu, A. and Ma殠and Bertini, E. and Keim, D. and Schreck, T. and Seidl, T.},
title = {{Subspace search and visualization to make sense of alternative clusterings in high-dimensional data}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
number = {1},
pages = {63--72},
year = {2012}
}
@article{Trimm2012,
author = {Trimm, D. and Rheingans, P. and DesJardins, M.},
doi = {10.1109/TVCG.2012.288},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2809--2818},
title = {{Visualizing student histories using clustering and composition}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327287},
volume = {18},
year = {2012}
}
@article{Walny2012,
author = {Walny, J. and Lee, B. and Johns, P. and Riche, N. H. and Carpendale, S.},
doi = {10.1109/TVCG.2012.275},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2779--2788},
title = {{Understanding pen and touch interaction for data exploration on interactive whiteboards}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327284},
volume = {18},
year = {2012}
}
@article{Wood2012,
author = {Wood, J. and Isenberg, P. and Isenberg, T. and Dykes, J. and Boukhelifa, N. and Slingsby, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2749--2758},
title = {{Sketchy rendering for information visualization}},
volume = {18},
year = {2012}
}
@article{Kang2012,
author = {Kang, Y. A. and Stasko, J. T.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2869--2878},
title = {{Examining the use of a visual analytics system for sensemaking tasks: Case studies with domain experts}},
volume = {18},
year = {2012}
}
@article{Ahmed2012,
author = {Ahmed, N. and Zheng, Z. and Mueller, K.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2104--2113},
title = {{Human computation in visualization: Using purpose driven games for robust evaluation of visualization algorithms}},
url = {http://www.cs.sunysb.edu/~mueller/papers/HPU\_vis\_2012.pdf},
volume = {18},
year = {2012}
}
@inproceedings{Andrews2012,
author = {Andrews, C. and North, C.},
title = {{Analyst's workspace: An embodied sensemaking environment For large, high-resolution displays}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {123--131},
year = {2012}
}
@inproceedings{vonLandesberger2012,
author = {Landesberger, T. von and Bremm, S. and Andrienko, N. and Andrienko, G. and Tekusova M.},
title = {{Visual analytics methods for categoric spatio-temporal data}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {183--192},
year = {2012}
}
@inproceedings{Cottam2012a,
author = {Cottam, J. A and Lumsdaine, A. and Weaver, C.},
title = {{Watch this: A taxonomy for dynamic data visualization}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {193--202},
year = {2012}
}
@article{Endert2012b,
author = {Endert, A. and Fiaux, P. and North, C.},
doi = {10.1109/TVCG.2012.260},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2879--2888},
title = {{Semantic interaction for sensemaking: Inferring analytical reasoning for model steering}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327294},
volume = {18},
year = {2012}
}
@inproceedings{Gou2012,
author = {Gou, L. and Zhang, X. L. and Luo, A. and Anderson, P. F.},
title = {{SocialNetSense: Supporting sensemaking of social and structural features in networks with interactive visualization}},
booktitle = {Proc. IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {133--142},
year = {2012}
}
@article{Kandel2012,
author = {Kandel, S. and Paepcke, A. and Hellerstein, J. M. and Heer, J.},
doi = {10.1109/TVCG.2012.219},
journal = {IEEE Transactions on Visualization and Computer Graphics},
number = {12},
pages = {2917--2926},
title = {{Enterprise data analysis and visualization: An interview study}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327298},
volume = {18},
year = {2012}
}
@article{Livingston2012,
author = {Livingston, M. A. and Decker, J. W. and Ai, Z.},
doi = {10.1109/TVCG.2012.223},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2114--2121},
title = {{Evaluation of multivariate visualization on a multivariate task}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327216},
volume = {18},
year = {2012}
}
@article{Pohl2012a,
author = {Pohl, M. and Smuc, M. and Mayr, E.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2908--2916},
title = {{The user puzzle: Explaining the interaction with visual analytics systems}},
volume = {18},
year = {2012}
}
@article{Sips2012,
author = {Sips, M. and Kothur, P. and Unger, A. and Hege, H. C. and Dransch, D.},
doi = {10.1109/TVCG.2012.191},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {12},
pages = {2899--2907},
title = {{A visual analytics approach to multiscale exploration of environmental time series}},
url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6327296},
volume = {18},
year = {2012}
}
@article{Bowman2012,
author = {Bowman, B. and Elmqvist, N. and Jankun-Kelly, T. J.},
doi = {10.1109/TVCG.2012.77},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {11},
pages = {1956--1968},
title = {{Toward visualization for games: Theory, design space, and patterns}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22392715},
volume = {18},
year = {2012}
}
@article{Kwon2012,
author = {Kwon, B. C. and Javed, W. and Ghani, S. and Elmqvist, N. and Yi, J. S. and Ebert, D.},
doi = {10.1109/TVCG.2012.89},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {11},
pages = {1992--2004},
title = {{Evaluating the role of time in investigative analysis of document collections}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22411889},
volume = {18},
year = {2012}
}
@article{Sajadi2012,
author = {Sajadi, B. and Majumder, A. and Oliveira, M. M. and Schneider, R. G. and Raskar, R.},
doi = {10.1109/TVCG.2012.93},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
title = {{Using patterns to encode color information for dichromats}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22450823},
volume = {17},
year = {2012}
}
@article{Tak2012,
author = {Tak, S. and Cockburn, A.},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
pages = {1--8},
title = {{Enhanced spatial stability with Hilbert and Moore treemaps}},
year = {2012}
}
@article{Eichelbaum2012,
author = {Eichelbaum, S. and Hlawitschka, M. and Scheuermann, G.},
doi = {10.1109/TVCG.2012.142},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
number = {1},
title = {{LineAO - Improved three-dimensional line rendering}},
url = {http://www.ncbi.nlm.nih.gov/pubmed/22689079},
volume = {6},
year = {2012}
}
@inproceedings{Dillingham2012,
author = {Dillingham, I. and Dykes, J. and Wood, J.},
title = {{A design, analysis and evaluation model to support the visualization designer-user}},
booktitle = {Posters IEEE Conf. Information Visualization (InfoVIs)},
year = {2012}
}
@inproceedings{Perin2012,
author = {Perin, C. and Vernier, F. and Fekete, J. D.},
title = {{Progressive horizon graphs: Improving small multiples visualization of time series}},
booktitle = {Posters IEEE Conf. Information Visualization (InfoVIs)},
year = {2012}
}
@inproceedings{Fisher2012,
author = {Calderon, N. A. and Riecke, B. E. and Fisher, B.},
title = {{Augmenting visual representation of affectively charged information using sound graphs}},
booktitle = {Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {113--114},
year = {2012}
}
@inproceedings{Green2012a,
author = {Green, T. M. and Fisher, B.},
title = {{Using translational science in visual analytics}},
booktitle = {Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {121--122},
year = {2012}
}
@inproceedings{Harrison2012,
author = {Harrison, L. and Chang, R. and Lu, A.},
title = {{Exploring the impact of emotion on visual judgement}},
booktitle = {Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {127--128},
year = {2012}
}
@inproceedings{Schwabish2012,
author = {Schwabish, J. A.},
title = {{Infographics at the Congressional Budget Office}},
booktitle = {Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {141--142},
year = {2012}
}
@inproceedings{Vadlapudi2012,
author = {Vadlapudi, R. and Siahbani, M. and Sarkar, A. and Dill, J.},
title = {{LensingWikipedia: Parsing text for the interactive visualization of human history}},
booktitle = {Posters IEEE Symp. Visual Analytics Science and Technology (VAST)},
pages = {147--148},
year = {2012}
}
}}}
{{{
@article{Andrews2013,
author = {Andrews, C. and North, C.},
number = {12},
pages = {2207--2216},
title = {{The impact of physical navigation on Spatial organization for sensemaking}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Andrienko2013,
author = {Andrienko, N. and Andrienko, G. and Barrett, B. and Dostie, M. and Henzi, P.},
number = {12},
pages = {2169--2178},
title = {{Space transformation for understanding group movement}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Bernard2013,
author = {Bernard, J. and Wilhelm, N. and Kr岬 B. and May, T. and Schreck, T. and Kohlhammer, J.},
number = {12},
pages = {2257--2266},
title = {{MotionExplorer: Exploratory search in human motion capture data based on hierarchical aggregation}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Bogl2013,
author = {B쬠M. and Aigner, W. and Filzmoser, P. and Lammarsch, T. and Miksch, S. and Rind, A.},
number = {12},
pages = {2237--2246},
title = {{Visual analytics for model selection in time series analysis}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Bosch2013,
author = {Bosch, H. and Thom, D. and Heimerl, F. and P䭡nn, E. and Koch, S. and Kr岬 R. and W, M. and Ertl, T.},
number = {12},
pages = {2022--2031},
title = {{ScatterBlogs2: Real-time monitoring of microblog messages through user -guided filtering}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Broeksema2013,
author = {Broeksema, B. and Baudel, T. and Telea, A. and Crisafulli, P.},
number = {12},
pages = {1972--1981},
title = {{Decision exploration lab: A visual analytics solution for decision management}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Choo2013,
author = {Choo, J. and Lee, C. and Reddy, C. K. and Park, H.},
number = {12},
pages = {1992--2001},
title = {{UTOPIAN: User-driven topic modeling based on interactive nonnegative matrix factorization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Dou2013a,
author = {Dou, W. and Yu, L. and Wang, X. and Ma, Z. and Ribarsky, W.},
number = {12},
pages = {2002--2011},
title = {{HierarchicalTopics: Visually exploring large text collections using topic hierarchies}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Ferreira2013,
author = {Ferreira, N. and Poco, J. and Vo, H. T. and Freire, J. and Silva, C. T.},
number = {12},
pages = {2149--2158},
title = {{Visual exploration of big spatio-temporal urban data: A study of New York City taxi trips}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Ghani2013,
author = {Ghani, S. and Kwon, B. C. and Lee, S. and Yi, J. S. and Elmqvist, N.},
number = {12},
pages = {2032--2041},
title = {{Visual anlytics for multimodal social network analysis: A design study with social scientists}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Gleicher2013a,
author = {Gleicher, M.},
number = {12},
pages = {2042--2051},
title = {{Explainers: Expert explorations with crafted projections}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Hadlak2013,
author = {Hadlak, S. and Schumann, H. and Cap, C. H. and Wollenberg, T.},
number = {12},
pages = {2267--2276},
title = {{Supporting the visual analysis of dynamic networks by clustering associated temporal attributes}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Hajizadeh2013,
author = {Hajizadeh, A. H. and Tory, M. and Leung, R.},
number = {12},
pages = {2189--2197},
title = {{Supporting awareness through collaborative brushing and linking of tabular data}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Hu2013,
author = {Hu, X. and Bradel, L. and Maiti, D. and House, L. and North, C. and Leman, S.},
number = {12},
pages = {2052--2059},
title = {{Semantics of directly manipulating spatializations}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Kodagoda2013,
author = {Kodagoda, N. and Attfield, S. and Wong, B. L. W. and Rooney, C. and Choudhury, S. T.},
number = {12},
pages = {2217--2226},
title = {{Using interactive visual reasoning to support sense-making: Implications for design}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Kurzhals2013,
author = {Kurzhals, K. and Weiskopf, D.},
number = {12},
pages = {2129--2138},
title = {{Space-time visual analytics of eye-tracking data for dynamic stimuli}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Legg2013,
author = {Legg, P. A. and Chung, D. H. S. and Parry, M. L. and Bown, R. and Jones, M. W. and Griffiths, I. W. and Chen, M.},
number = {12},
pages = {2109--2118},
title = {{Transformation of an uncertain video search pipeline to a sketch-based visual analytics loop}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Meghdadi2013,
author = {Meghdadi, A. H. and Irani, P.},
number = {12},
pages = {2119--2128},
title = {{Interactive exploration of surveillance video through action shot summarization and trajectory visualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Monroe2013b,
author = {Monroe, M. and Lan, R. and Lee, H. and Plaisant, C. and Shneiderman, B.},
number = {12},
pages = {2227--2236},
title = {{Temporal event sequence simplification}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Polishchuk2013,
author = {Packer, E. and Bak, P. and Nikkilͮ and Polishchuk, V. and Ship, H. J.},
number = {12},
pages = {2179--2188},
title = {{Visual analytics for spatial clustering: Using a heuristic approach for guided exploration}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Rind2013,
author = {Rind, A. and Lammarsch, T. and Aigner, W. and Alsallakh, B. and Miksch, S.},
number = {12},
pages = {2247--2256},
title = {{TimeBench: A data model and software library for visual analytics of time-oriented data}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Schmidt2013,
author = {Schmidt, J. and Gr쥲, M. E. and Bruckner, S.},
number = {12},
pages = {2090--2099},
title = {{VAICo: Visual analysis for image comparison}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Schultz2013,
author = {Schultz, T. and Kindlmann, G. L.},
number = {12},
pages = {2100--2108},
title = {{Open-box spectral clustering: Applications to medical image analysis}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Shadoan2013,
author = {Shadoan, R. and Weaver, C.},
number = {12},
pages = {2070--2079},
title = {{Visual analysis of higher-order conjunctive relationships in multidimensional data using a hypergraph query system}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Thomas2013,
author = {Thomas, M. and Piringer, H.},
number = {12},
pages = {1962--1971},
title = {{A partition-based framework for building and validating regression models}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Walker2013,
author = {Walker, R. and Slingsby, A. and Dykes, J. and Xu, K. and Wood, J. and Nguyen, P. H. and Stephens, D. and Wong, B. L. W. and Zheng, Y.},
number = {12},
title = {{An extensible framework for provenance in human terrain visual analytics}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Wang2013a,
author = {Wang, B. and Ruchikachorn, P. and Mueller, K.},
number = {12},
pages = {2060--2069},
title = {{SketchPad N-D: WYDIWYG sculpting and editing in + LJK dimensional space}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Wang2013,
author = {Wang, Z. and Lu, M. and Yuan, Y. and Zhang, J. and de Wetering, H. v.},
number = {12},
pages = {2159--2168},
title = {{Visual traffic jam analysis based on trajectory data}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Willett2013,
author = {Willett, W. and Ginosar, S. and Steinitz, A. and Hartmann, B. and Agrawala, M.},
number = {12},
pages = {2198--2206},
title = {{Identifying redundancy and exposing provenance in crowdsourced data analysis}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Xu2013,
author = {Xu, P. and Wu, Y. and Wei, E. and Peng, T. Q. and Liu, S. and Zhu, J. J. H. and Qu, H.},
number = {12},
pages = {2012--2021},
title = {{Visual analysis of topic competition on social media}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Zhang2013,
author = {Zhang, J. and Kang, K. and Liu, D. and Yuan, Y. and Y., E.},
number = {12},
pages = {1982--1991},
title = {{Vis4Heritage: Visual analytics approach on grotto wall painting degradations}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Zhao2013,
author = {Zhao, J. and Collins, C. and Chevalier, F. and Balakrishnan, R.},
number = {12},
pages = {2080--2089},
title = {{Interactive exploration of implicit and explicit relations in faceted datasets}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Alsallakh2013,
author = {Alsallakh, B. and Aigner, W. and Miksch, S. and Hauser, H.},
number = {12},
pages = {2496--2505},
title = {{Radial sets: Interactive visual analysis of large overlapping sets}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Bakke2013,
author = {Bakke, E. and Karger, D. R. and Miller, R. C.},
number = {12},
pages = {2586--2595},
title = {{Automatic layout of structured hierarchical reports}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Basole2013,
author = {Basole, R. C. and Clear, T. and Hu, M. and Mehrotra, H. and Stasko, J. T.},
number = {12},
pages = {2526--2535},
title = {{Understanding interfirm relationships in business ecosystems with interactive visualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Borkin2013,
author = {Borkin, M. A. and Vo, A. A. and Bylinskii, Z. and Isola, P. and Sunkavalli, S. and Oliva, A. and Pfister, H.},
number = {12},
pages = {2306--2315},
title = {{What makes a visualization memorable?}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Borkin2013a,
author = {Borkin, M. A. and Yeh, C. .S and Boyd, M. and Macko, P. and Gajos, K. Z. and Seltzer, M. and Pfister, H.},
number = {12},
pages = {2476--2485},
title = {{Evaluation of filesystem provenance visualization tools}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Brehmer2013a,
author = {Brehmer, M. and Munzner, T.},
number = {12},
pages = {2376--2385},
title = {{A multi-level typology of abstract visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Chevalier2013,
author = {Chevalier, F. and Vuillemot, R. and Gali, G.},
number = {12},
pages = {2426--2435},
title = {{Using concrete scales: A practical framework for effective visual depiction of complex measures}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Dwyer2013,
author = {Dwyer, T. and Riche, N. H. and Marriott, K. and Mears, C.},
number = {12},
pages = {2596--2605},
title = {{Edge compression techniques for visualization of dense directed graphs}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Ferstay2013,
author = {Ferstay, J. A. and Nielsen, C. B. and Munzner, T.},
number = {12},
pages = {2546--2555},
title = {{Variant View: Visualizing sequence variants in their gene context}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Fink2013,
author = {Fink, M. and Haunert, J. H. and Spoerhase, J. and Wolff, A.},
number = {12},
pages = {2326--2335},
title = {{Selecting the aspect ratio of a scatter plot based on its Delaunay triangulation}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Gleicher2013,
author = {Gleicher, M. and Correll, N. and Nothelfer, C. and Franconeri, S.},
number = {12},
pages = {2316--2325},
title = {{Perception of average value in multiclass scatterplots}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Goodwin2013a,
author = {Goodwin, S. and Dykes, J. and Jones, S. and Dillingham, I. and Dove, G. and Duffy, A. and Kachkaev, A. and Slingsby, A. and Wood, J.},
number = {12},
pages = {2516--2525},
title = {{Creative user-centered visualization design for energy analysts and modelers}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Gratzl2013,
author = {Gratzl, S. and Lex, A. and Gehlenborg, N. and Pfister, H. and Streit, M.},
number = {12},
pages = {2277--2286},
title = {{LineUp: Visual analysis of multi-attribute rankings}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Guerra-gomez2013,
author = {Guerra-G庬 J. A. and Pack, M. L. and Plaisant, C. and Shneiderman, B.},
number = {12},
pages = {2566--2575},
title = {{Visualizing change over time using dynamic hierarchies: TreeVersity2 and the StemView}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Hofmann2013,
author = {Hofmann, H. and Vendettuoli, M.},
number = {12},
pages = {2297--2305},
title = {{Common angle plots as perception-true visualizations of categorical associations}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Hullman2013a,
author = {Hullman, J. and Drucker, S. and Riche, N. H. and Lee, B. and Fisher, D. and Adar, E.},
number = {12},
pages = {2406--2415},
title = {{A deeper understanding of sequence in narrative visualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Huron2013,
author = {Huron, S. and Vuillemot, R. and Fekete, J. D.},
number = {12},
pages = {2446--2455},
title = {{Visual sedimentation}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Mcguffin2013,
author = {Im, J. F. and McGuffin, M. J. and Leung, R.},
number = {12},
pages = {2606--2614},
title = {{GPLOM: The generalized plot matrix for visualizing multidimensional multivariate data}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Isenberg2013a,
author = {Isenberg, P. and Dragicevic, P. and Willett, W. and Bezerianos, A. and Fekete, J. D.},
number = {12},
pages = {2346--2355},
title = {{Hybrid-image visualization for large viewing environments}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Jakobsen2013,
author = {Jakobsen, M. R. and Haile, Y. S. and Knudsen, S. and Hornb젋.},
number = {12},
pages = {2386--2395},
title = {{Information visualization and proxemics: Design opportunities and empirical findings}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Jakobsen2013a,
author = {Jakobsen, M. R. and Hornb젋.},
number = {12},
pages = {2336--2345},
title = {{Interactive visualizations on large and small displays: The interrelation of display size, information space, and scale}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Jansen2013,
author = {Jansen, Y. and Dragicevic, P.},
number = {12},
pages = {2396--2405},
title = {{An interaction model for visualizations beyond the desktop}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Kehrer2013,
author = {Kehrer, J. and Piringer, H. and Berger, W. and Gr쥲, M. E.},
number = {12},
title = {{A model for structure-based comparison of many categories in small-multiple displays}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Lee2013,
author = {Lee, B. and Kazi, R. H. and Smith, G.},
number = {12},
pages = {2416--2425},
title = {{SketchStory: Telling more engaging stories with data through freeform sketching}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Lehmann2013,
author = {Lehmann, D. J. and Theisel, H.},
number = {12},
title = {{Orthographic star coordinates}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Lex2013,
author = {Lex, A. and Partl, C. and Kalkofen, D. and Streit, M. and Gratzl, S. and Wassermann, A. M. and Schmalstieg, D. and Pfister, H.},
number = {12},
pages = {2536--2545},
title = {{Entourage: Visualizing relationships between biological pathways using contextual subsets}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Lins2013,
author = {Lins, L. and Klosowski, J. T. and Scheidegger, C.},
number = {12},
pages = {2456--2465},
title = {{Nanocubes for real-time exploration of spatiotemporal datasets}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Liu2013a,
author = {Liu, S. and Wu, Y. and Wei, E. and Liu, M. and Liu, Y.},
number = {12},
pages = {2436--2445},
title = {{StoryFlow: Tracking the evolution of stories}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Maguire2013,
author = {Maguire, E. and Rocca-Serra, P. and Sansone, S. A. and Davies, J. and Chen, M.},
number = {12},
pages = {2576--2585},
title = {{Visual compression of workflow visualizations with automated detection of macro motifs}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Perin2013a,
author = {Perin, C. and Vuillemot, R. and Fekete, J. D.},
number = {12},
pages = {2506--2515},
title = {{SoccerStories: A kick-off for visual soccer analysis}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Roth2013a,
author = {Roth, R. E.},
number = {12},
pages = {2356--2365},
title = {{An empirically-derived taxonomy of interaction primitives for interactive cartography and geovisualization}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Mcguffin2013a,
author = {Rufiange, S. and McGuffin, M. J.},
number = {12},
pages = {2556--2565},
title = {{DiffAni: Visualizing dynamic graphs with a hybrid of difference maps and animation}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Sambasivan2013,
author = {Sambasivan, R. R. and Shafer, I. and Mazurek, M. L. and Ganger, G. R.},
number = {12},
pages = {2466--2475},
title = {{Visualizing request-flow comparison to aid performance diagnosis in distributed systems}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Schulz2013,
author = {Schulz, H. J. and Nocke, T. and Heitzler, M. and Schumann, H.},
number = {12},
pages = {2366--2375},
title = {{A design space of visualization tasks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Sedlmair2013a,
author = {Sedlmair, M. and Munzner, T. and Tory, M.},
number = {12},
pages = {2634--2643},
title = {{Empirical guidance on scatterplot and dimension reduction technique choices}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Taimouri2013,
author = {Taimouri, V. and Hua, J.},
number = {12},
pages = {2644--2652},
title = {{Visualization of shape motions in shape space}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Vehlow2013,
author = {Vehlow, C. and Reinhardt, T. and Weiskopf, D.},
number = {12},
pages = {2486--2495},
title = {{Visualizing fuzzy overlapping communities in networks}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
@article{Yuan2013,
author = {Yuan, X. and Ren, D. and Wang, Z. and Guo, G},
number = {12},
pages = {2625--2633},
title = {{Dimension projection matrix/tree: Interactive subspace visual exploration and analysis of high dimensional data}},
journal = {IEEE Trans. Visualization and Computer Graphics (TVCG)},
volume = {19},
year = {2013}
}
}}}
<<list filter [tag[vismon]]>>
!!Logistics: short-term
*Is the February Vismon workshop still going ahead as planned?
**We need information ASAP re: ''dates'' and ''times'', ''location'', ''people'' attending, ''key players'', meeting ''agenda''
**Should an HCI (~Human-Computer Interaction) observer be on the ground? What are your expectations? (see Evaluation section below)
***We need to make arrangements ASAP
!!Logistics: long-term
#What is the staged ''deployment timeline'' for Vismon?
#How many user groups are involved?
##Where are these user groups located?
##How large are these user groups?
##Will there be a Vismon workshop for each user group? When / where are these set to occur?
#How do the user groups differ?
##Are all user groups expected to use a ''single codebase'', or will there be ''multiple deployments'' to suit the needs of the different user groups?
##Will all groups be making use of the same simulation data format? Are management options and indicators consistent?
#What is the ''long-term development plan'' of Vismon?
##When will maintenance and upgrades occur?
##Will there be follow-up workshops? When / how often?
!!Clarification
*We need a better understanding of ''Vismon's intended role'' in the workflow of fisheries managers and policymakers:
#What is their ''current workflow''?
#How are policy decisions currently made?
##How much evidence is currently provided to managers and policy makers vs. how much will Vismon provide?
#With regards to both simulation and visualization, what are users' ''current levels of understanding''?
##How much does this vary within and between groups?
##What is your distinction between a ''technical'' and ''non-technical'' user?
#Is Vismon intended to ''replace'' part or all of an existing workflow, to ''supplement'' an existing workflow, or does it ''add additional'' steps to an existing workflow?
#Is the use of Vismon ''mandated'', or is it intended to be used at the discretion of the individual user (optional usage)?
#What is the current ''workflow of fisheries scientists''?
##How much of this workflow is being offloaded to Vismon for managers and policymakers to perform?
##What parts of scientists' workflow are being retained?
#To what extent were Vismon's ''intended users involved in the development process'' thus far?
!!Evaluation
*How do you plan to document ''success''?
**What is the ''desired outcome'' of the February workshop?
**What is the ''desired outcome'' of the project in its entirety?
*What would you consider a negative result or failure?
!!!Focus / scope of study
*Given what we already know about the project, we have several research questions. Which questions we will address will depend on the project timeline, and your own evaluation criteria.
*Since Vismon is both a simulation and visualization tool, we will maintain a separation between these two aspects of the tool. What is the ''relative contribution of simulation and visualization'' to the following:
#''Utility / Efficacy'':
##Does use of Vismon result in better policy decisions?
#''Usability'':
##When using Vismon, which tasks are difficult or impossible to perform?
##Which tasks are inefficiently performed?
##What design features contribute to these problems?
#''Learnability'':
##What is the learning curve of Vismon?
##Are the instructional workshops and training materials effective?
##How can Vismon be made easier to learn?
#''Adoption'': (depends on whether adoption of Vismon is mandated, and whether it is intended to replace, supplement, or add elements to users' workflows):
##How often is Vismon used?
##When is Vismon used?
##How does Vismon fit into the workflow of its users?
##How does it affect collaboration, communication (between policymakers, between policymakers and scientists, policymakers and other stakeholders)?
##If use of the tool is discontinued, what are the causes?
*How should these foci be prioritized given your evaluation criteria?
!!Methods
Depending on the scope and timeline of the study (shallow longitudinal vs. deep longitudinal), some combination of the following research methods may be used:
*''User observation'' (at workshops vs. during a typical workflow)
*One-on-one ''interviews'' with users: initial interviews at workshops + follow-up interviews via phone or video-conference
*''Contextual inquiry'': analysis of the user behaviour in the context of their work environment, their high-level workflow and responsibilities, collaboration and communication with others
*''Focus groups'': with users, users and scientists, Vismon development team, other stakeholders
*''Analysis of artifacts'' and documents produced by users: usage logs, bug reports, journals, reports, communication
*''Questionnaire'' responses
!!Why we want to be involved:
*A ''unique situation'' in which a tool has been developed in collaboration with one group (fisheries scientists), but intended for use by another group (fisheries managers / policymakers). The extent to which the latter group was involved in Vismon's development process is unclear to us.
*The research community is ''in need of post-deployment studies''
*An opportunity to determine the ''efficacy of different evaluation methodologies'' and techniques
A set of design /evaluation heuristics for [[Information Visualization]] tools: Overview first, zoom and filter, provide details on demand, relate and extract information, and provide a history of interactions. While not high-level- or domain task-specific, these heuristics pertain largely to the usability of visualization tools. They neither directly relate to low-level perceptual and cognitive tasks.
Source: <<cite Shneiderman1996 bibliography:Bibliography-ToReadPriority>>
!!Data repositories / resources
*[[enigma.io|http://enigma.io/]] curated public data repository
*[[dreamtolearn repository list|https://dreamtolearn.com/node/2HUWRTIROI1VEB4B2FHE4ELN6/]]
*[[VAST challenge data|http://vacommunity.org/VAST+Challenge+2014]]
*[[Stats Canada|http://www.statcan.gc.ca]] (and similar agencies in other countries / jurisdictions)
!!Inspiration: interactive visualization survey sites
*[[keyvis.org|http://www.keyvis.org/]] - browse visualization papers by journal/conference paper keywords
!!!By data abtraction
*[[tree vis survey|http://treevis.net/]]
*[[set vis survey|http://www.setviz.net/]]
*[[text vis survey|http://textvis.lnu.se/]]
*[[time vis survey|http://timeviz.net/]]
*[[space-time cube vis survey|http://spacetimecakes.saclay.inria.fr/WebContent/index.php]]
!!!By domain
*[[performance vis|http://idav.ucdavis.edu/~ki/STAR/]]
*[[finance vis|http://financevis.net/]]
*[[wikipedia vis|http://seealso.org/]]
!!Inspiration: recent books
*[[Visualizing Data's reference list|http://www.visualisingdata.com/index.php/references/]]
*[[Design for Information|http://www.amazon.ca/dp/1592538061]] by Isabel Meirelles
*[[The Book of Trees|http://www.amazon.ca/dp/1616892188]] and [[Visual Complexity|http://www.amazon.ca/dp/1568989369]] by Manuel Lima
*[[The Functional Art|http://www.amazon.ca/dp/0321834739]] by Alberto Cairo
*[[Visual Insights|http://www.amazon.ca/dp/0262526190]] by Katy B沮er and David Polley
!!Inspiration:
*@mattbrehmer's List of [[100+ visualization practitioners/designers on twitter|https://twitter.com/mattbrehmer/lists/infovis]]
!!Data wrangling / cleaning / management / EDA
*[[dat|http://dat-data.com/]] - data package management (new!)
*[[Wrangler|http://vis.stanford.edu/wrangler/]]
*[[R, RStudio|http://www.rstudio.com/]] + [[ggplot2|http://ggplot2.org/]]
**learn R: [[UBC STAT 545 resources on github|http://stat545-ubc.github.io/]] ([[2013 course materials|https://github.com/jennybc/STAT545A_2013]])
**learn R: [[Jeff Leek's Computing for Data Analysis course materials|https://github.com/jtleek/dataanalysis]]
*[[pandas|http://pandas.pydata.org/]] - python data analysis library; [[matplotlib|http://matplotlib.org/]] - python plotting
*[[Matlab|http://www.mathworks.com/products/matlab/]]
!!Visualization design
*[[datavisualization.ch|http://datavisualization.ch/]] survey site
*[[Visualizing Data's resource list|http://www.visualisingdata.com/index.php/resources/]]
!!!Mockups and wireframing
*[[Balsamiq|http://balsamiq.com/products/mockups/]] ($80 USD)
*[[Axure|http://www.axure.com/free-software-for-students]] (free for students)
*[[OmniGraffle|https://store.omnigroup.com/edu/24197f4053890389ffffffff/]] ($60 USD for students)
!!!Vis design (no programming)
*[[Tableau Desktop|http://www.tableausoftware.com/academic/students]] (free for students, ability to publish visualizations online)
**[[Tableau tutorials|http://www.tableausoftware.com/learn/training]]
*[[Gephi|https://gephi.github.io/]] for graph visualization
*[[NodeXL|http://nodexl.codeplex.com/]] - graph visualization template for Excel
*[[Lyra|http://idl.cs.washington.edu/projects/lyra/]] visualization design environment
*[[ManyEyes|http://www-958.ibm.com/software/analytics/manyeyes/]]
!!!Vis design (programming)
!!!!D3.js
*[[D3.js|http://d3js.org/]]
**learn D3: [[Interactive Data Visualization for the Web|http://chimera.labs.oreilly.com/books/1230000000345]] by Scott Murray (O'Reilly free ebook)
**learn D3: [[Dashing D3.js online tutorial|https://www.dashingd3js.com/table-of-contents]]
**learn D3: [[bl.ocks/org/mbostock|http://bl.ocks.org/mbostock]]
*[[Cubism.js|https://square.github.io/cubism/]] - D3 plugin for visualizing time series
*[[Crossfilter.js|http://square.github.io/crossfilter/]] - D3 plugin for multivariate datasets
!!!!Processing
*[[Processing|http://processing.org/]]
*[[Processing.js|http://processingjs.org/]]
*[[p5.js|http://p5js.org/]] ([[hello.p5js|http://hello.p5js.org/]]) (new!)
*[[Handy|http://www.gicentre.net/handy/]] - Hand-drawn sketchy rendering in Processing
**learn Processing: [[Visualizing Data|http://www.amazon.ca/dp/0596514557]] by Ben Fry
!!!!R
*[[Shiny|http://shiny.rstudio.com/]]
*[[ggvis|http://ggvis.rstudio.com/]]
*[[rCharts|http://ramnathv.github.io/rCharts/]]
*[[animint|http://sugiyama-www.cs.titech.ac.jp/~toby/animint/index.html]]: R package provides limited interactivity and animation for charts
*[[G3Plot|http://glimmer.rstudio.com/alexbbrown/g3plot/#dataSet=sunspots,tabSelected=G3Plot]]
*[[rCharts + D3|http://zevross.com/blog/2014/04/03/interactive-visualization-from-r-to-d3-using-rcharts/]]
!!!!Python
*[[bokeh|http://bokeh.pydata.org/]]
*[[iPython notebook|http://ipython.org/notebook.html]]
!!!!Basic charts
*[[Highcharts.js|http://www.highcharts.com/]]
*[[Polychart.js|http://polychart.com/]] (built w/ D3.js)
*[[NVD3|http://nvd3.org/]] (built w/ D3.js)
*[[Rickshaw|http://code.shutterstock.com/rickshaw/]] (time-series charts, built w/ D3.js)
!!Showing off your work
*[[Shiny Server|http://www.rstudio.com/products/shiny/shiny-server/]]
*[[bl.ocks.org|http://bl.ocks.org/]] - a viewer for [[Github Gist|https://gist.github.com/]]
*[[visualizing.org|http://visualizing.org]]
!Measuring --Validity-- Accuracy in Annotation: Humans and Machines Learning Together
[[Dr. Stuart Shulman|http://people.umass.edu/stu]], Jan. 19 2012 - Workshop sponsored by SLAIS (UBC School of Library, Archival and Information Studies) and the [[GRAND NCE|http://grand-nce.ca/research/opportunities-funding-training-employment/workshops-and-symposia/workshop-on-text-and-social-media-analysis]].
>//...a wealth of information creates a poverty of attention.// - Herbert Simon, 1971
Don't be dogmatic about methods: qualitative vs. quantitative. Regardless:
*Maintain replicability, be transparent,
*Pay close attention to error, have corrective measures
*Internal and External validation of results
*Don't rely solely on software, you must have justification for all the tools you choose
Working with qualitative data:
*''Purists'' - immersion, antipathy to numbers, in-depth analysis, contextual, subjective
*''Pluralists'' - experimental mixed method adaptive approach, flexible, interdisciplinary
*''Positivists'' - quantitative focus on error measurement, critical validity, reliability, replication, objectivity, generalization
What are the ''emergent properties'' of a body of text? What is the ''landscape'' of the text?
*Once you have found, coded, and tagged these properties, how do you achieve credibility in your research?
*How do you maintain inter-rater reliability?
Current work draws from the DARPA GALE project: Global Autonomous Language Exploitation
[[CAT|http:/cat.ucsur.pitt.edi]] - ''Coding Analysis Toolkit'' (free, open-source), can import raw text, import from ATLAS.ti
*easily measure inter-coder reliability, team coding
*focus on text (not yet images)
*soft-bounded vs. hard-bounded codes; how do you resolve / set tolerance for soft codes?
*functionality for adjudication, resolving inter-coder differences, merge and reduce number of codes, individual coder performance reports, confidence - how do you know when you've overtrained your coders?
**reduce to a set of codes to a point that you can train a machine classifier
[[DiscoverText|http://discovertext.com/]] - commercial version of earlier work, survey text analysis, social media monitoring, public opinion, general eDiscovery.
*Iterative search, classify, report, filter, (repeat)
*de-duplication, near-duplication removal for public comments, twitter
*annotation: applying human interpretation to machine classification
*Import from .zip, FDMS, Excel, CSV, Survey Analytics, XML, Facebook, Twitter, RSS/Atom, Google+, Google Reader, ~YouTube ~InfoExtractor, ~GovTrack Data, GNIP ~PowerTrack
*Likely to become a platform, allow integration of custom machine classifier ~APIs
*Secure, running in the cloud
Laura Macnamara, Sandia National Labs
*formative design, cognitive work analysis, cognitive task analysis, work domain analysis
*Green, Wakkary, Fisher, ~Arias-Hernandez HICSS 2011
*Ecological psychology a precedent for CWA.
*Neelam Naikar's "Work Domain Analysis", Vicente's CWA.
*Abstraction-decomposition: stuff / resources, processes, functions, metrics, purpose
*Applied CTA - Laura Militello and Hutton - Ergnomics 41 (1998)
*Working Minds - Crandall, Klein, Hoffman
*Handbook of Task Analysis for HCI.
*Handbook of Human Factors and Ergonomic Methods
*Neville Stanton's flow chart for Hierarchical Task Analysis (HTA)
*Cognitive task analysis is sequential
*Snag-it screen capture software.