Considerations for Interactive & Expressive Information Design Tools

Matthew Brehmer · EPIC Research Group, Microsoft Research · @mattbrehmer

Presentation at Autodesk Research 2019-02-26 · slides: mattbrehmer.github.io#talks

Outline

· My background and methods

· Considerations for expressive information design

· Timeline Storyteller

· Other recent tools: ChartAccent, Charticulator, & DataToon

· Opportunities for future research

EPIC: Extended Perception, Interaction & Cognition

My Background

2011 - 2016: PhD Computer Science specializing in Information Visualization, UBC

2009 - 2011: MSc Computer Science specializing in Human-Computer Interaction, UBC

2004 - 2009: Bachelor of Computing specializing in Cognitive Science, Queen's Univ.

Scope of My Research

Considerations and tools for expressive information design:
· Timeline Storyteller ^{(C+J 19)} | DataToon ^{(CHI 19)} | Charticulator ^{(TVCG 19)} | ChartAccent ^{(PVIS 17)} | Timelines Revisited ^{(TVCG 17)} | TimeLineCurator ^{(TVCG 16)}

Visualization task analysis:
· A Typology of Abstract Visualization Tasks ^{(TVCG 13)} | Visualizing Dimensionally-Reduced Data ^{(BELIV 14)}

Empirical evaluation:
· Data-Driven Stories ^{(DDS 18)} | Visualization Authoring Systems ^{(BELIV 18)} | Variants of Multi-Series Bar Charts ^{(CHI 18)} |
· Overview: A Document Mining Tool for Journalists ^{(TVCG 14)} | Pre-Design Empiricism for Visualization ^{(BELIV 14)}

Visualization for mobile devices:
· Ranges Over Time ^{(TVCG 19)} | Animation vs. Small Multiples ^{(in prep.)} | Novel Interactions with Time-Varying Data ^{(in prep.)}

Visualization for resource conservation:
· Workflows for Energy Portfolio Analysis ^{(TVCG 16)}

Scope of My Research

Design & Research Methods

Design & Implementation:

· User interface design | Visualization design & development | Toolkit development

Qualitative Research:

· Visualization design studies | Requirements analysis | Retrospective interviews
· Chauffeured demos | Content analysis | Post-deployment usage analysis

Quantitative Research:

· Laboratory experiments | Crowdsourced experiments | Statistical analysis

In terms of how I do my research,

I spend a lot of time designing and building things

I've alternated between visualization and interface design at various levels of fidelity,

as well as software development for a deployed product, a research toolkit, and various visualization tools and experimental applications.

Though just as important are the human-centred research methods that I use,

This includes quite a bit of qualitative evaluation work during my PhD in the context of visualization design or case studies,

Which involved everything from requirements analysis to analyzing the usage of deployed visualization tools.

This qualitative work also involves systematically collecting and categorized previous research as well as information design content done by practitioners.

And finally I also do lab-based experimental work, and especially recently with some experiments involving crowdsourcing,

I'll revisit some of these methods later in this talk when I speak about evaluating interactive visualization tools.

Democratizing Information Design

How can I enable under-served groups of people to...

Expressively visualize their data?

Produce and present compelling data-driven stories?

Applications of visualization beyond those in professional data analysis.

e.g., Why are journalists and educators presenting information using business intelligence tools?

With respect to the values and goals that underlie my research,

I'm really hoping to advance the democratization of information design and data visualization:

And this means enabling more people to visualize and present compelling stories about their data,

And its in this respect that I contrast my research from that of many others in the broader visualization community, such as those working in visual analytics.

Many in the visualization and visual analytics research community are motivated by use cases from professional data analysis, data science, or business intelligence.

While I think these use cases are important and I'm glad that people are thinking about them, they are NOT the use cases that motivate my research.

Instead, I'm interested in incorporating visualization to present information to people who don't identify as data analysts, in both public and personal contexts.

This leads me to ask questions like ``why are journalists and educators are using business intelligence tools to present information to their audiences``?

And I wonder if I build better tools for these communities and the needs that they have

Connecting Research & Practice

Disseminating visualization research into practice, and vice versa.

Promoting and studying the adoption of deployed information design tools and research prototypes.

Collecting examples of information design produced by practitioners.

Fostering a dialogue between researchers and practitioners (e.g., OpenVisConf, VisInPractice)*. * OpenVisConf: openvisconf.com | VisInPractice event at IEEE VIS: VisInPractice.github.io .

Another thing that drives my work is connecting visualization researchers and practitioners.

This means disseminating research into practice, and just as importantly, disseminating practice into research.

In my own work, this manifests as studying the adoption of deployed tools and prototypes.

And collecting examples of information design produced by practitioners.

And finally this means speaking directly with practitioners and presenting my work at conferences like OpenVisConf,which is one of the most renowned practitioner-oriented visualization events,

Or how I co-organized the VisInPractice event at last year's IEEE VIS conference, where I invited prominent visualization practitioners to speak to a largely academic audience.

Expressive Information Design

From the perspective of an information visualization researcher.

Expressive Information Design

· Combining visualization, annotation, and explanation to present information to an audience.

· Thinking systematically about tasks, design choices, and constraints.

· Identifying ways to assess alternative design choices.

Presenting Information to the Public

e.g., Hans Rosling's TED presentations about global economic and public health indicators.
Image: Open Knowledge Foundation Deutschland (flickr, cc by).

Presenting Information to the Individual

Image: newkemall (flickr, cc by).

Aspects of Expressive Information Design

Thinking systematically about tasks, design choices, and constraints.

Thinking Systematically about Tasks

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

A central theme of my PhD research pertained to classifying visualization tasks and to consolidating the vocabulary for visualization tasks into a typology,

Which is the subject of a 2013 paper written with my PhD advisor Tamara Munzner.

And expressive information design requires being able to address a wide range of tasks,

Including both the tasks of an information designer or author, as well as the tasks of a viewer, or what the audience is expected to do with the information presented to them.

Such as to identify a trend or to compare values.

Ultimately, expressivity is about satisfying the intent of the author, allowing them to tell the story they want to tell,

And whether this story can be reconstructed in the mind of the viewer.

Thinking Systematically about Tasks

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

The most cited IEEE InfoVis paper since 2013, with more than 280 citations*.

* Google Scholar, Jan 2019.

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

Thinking Systematically about Design Choices

Identifying the dimensions of design spaces that characterize:

... ways to visually represent data.

... ways to interact with these representations.

... ways to highlight and annotate them.

... ways to combine visual content with textual explanation.

In an expressive information design tool, how do you present these choices?

e.g., Low-level visual encoding choices (shape, color, ...) or chart templates (bar, line, ...)

Expressivity also refers to thinking systematically about design choices and design spaces:

Which are ways to categorize visual representation choices for a particular type of data,

Interaction design choices, ways to highlight, annotate, and embellish visual representations,

Or ways to combine visual and textual explanations in compelling and engaging ways.

Characterizing these design spaces becomes particularly important when you decide to expose them to an author in the form of an information design tool, where there is a question about the level of granularity:

For example, as a designer are you controlling low-level visual encoding channels like shape or color, or are you picking a chart type from a template?

As you'll see an example of this process of identifying and assessing points in a design space in the next section of this talk .

Constraints on Expressivity

Constraints imposed by authors, by the audience, or by the context.

Expertise: empowering non-programmers and non-designers.

Resources: enabling expressive information design for those on a deadline.

Literacy: anticipating the visual and data literacy of the audience.

Device: anticipating the audience's viewing experience.

But first I want to mention that expressivity is also bound to constraints, and it is these constraints that I find to be particularly challenging and deserving of research.

And these include constraints imposed by designers or authors, as well as those imposed by the audience and the context in which the information is presented.

<< CLICK >> One of these constraints is expertise:

I want to empower non-programmers to present information with interactive visualization and presentation tools.

And I want to empower non-designers to make compelling and useful information graphics by offering design guidance.

<< CLICK >> Another constraint is about resources:

For example, I want to enable over-committed, understaffed, and poorly funded local news teams to produce data-driven stories about their community while meeting all their deadlines.

<< CLICK >> Yet another constraint is literacy:

What if my audience lacks the sufficient visual literacy or data literacy needed to interpret my particular choice of visual representation?

How can I make use of their prior knowledge as part of the communication process?

<< CLICK >> And finally there's the device: what if my audience is consuming this information from a mobile device?

How does that constrain my choices?

Outline

The Daily Routines of Famous Creative People

Story inspired by infographics by Podio and info we trust.
Data source: Daily Rituals: How Artists Work by Mason Currey (2013)

To illustrate this, I want to tell you a few interesting things about the daily routines of famous creative people.

MAX: What you see here is a set of radial timelines depicting a typical 24 hours in the lives of 26 writers, artists, composers, and other creative types.

When they work, eat, sleep, exercise, and do other activities.

NEXT: I had wondered if there was any pattern as to when creative people do their creative work.

It turns out that at least among these people, some creative people prefer to work in short, focused bursts, like Darwin or Kant, while people like Murakami and Voltaire get up early and work for long uninterrupted periods of time.

Meanwhile, you have odd ones like Kafka who work all night long.

NEXT: I had also wondered about the potential relationship between sleep and creativity, so let me focus your attention to these blue arcs.

Most people here tend to sleep at night, with notable exceptions like Balzac who would turn in at 6pm, and Kafka, who just seemed to keep odd hours.

A third question I had was about variation and creativity. Let me show you how often these creative people change activities throughout their day.

NEXT: This 24-hour clock metaphoe isn't the best way to convey the number or heterogeneity of activities, so I'm going to transition to a just show you a sequence of activities without any duration, effectively resulting in a radial bar chart.

NEXT: To show who varied the most and least among these people, a linear representation is probably better, where you can see that Darwin has the most varied day of all the people here while Murakami prefers less variation.

NEXT: Finally, I'll restore the duration of events so that you can compare these timelines just by scanning up and down, where you might spot times where some routines appear to be in sync with one another, such as when people work or sleep at the same time.

And this is a good time to ask you to compare your own daily rhythm to these creative people.

When do you sleep, eat, or do you best creative work?

Expressive Information Design with Timelines

Timeline Storyteller: The Design & Deployment of an Interactive Authoring Tool for
Expressive Timeline Narratives.
Brehmer, Lee, Henry Riche, Tittsworth, Lytvynets, Edge, and White. In Proc. Comp. + Journalism 2019.

timelinestoryteller.com | github.com/Microsoft/timelinestoryteller

Timelines Revisited

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

Timelines are visual representations of categorical event sequences.

How have people drawn timelines over the course of history?

The visualization research community has focused on their use in data analysis.

How have practitioners used them for storytelling?

What Happened When?

In what sequence did the events occur?

How long did the events last?

How long between event A and event B?

Did A and B co-occur or repeat?

When did A and B occur relative to event C?

A Timeline Design Space

Scale

Layout
Unified

It turns out that there are many ways to answer these questions and present timelines, which brings me to my design space.

You can represent time in various ways, such as linearly, radially, or by using spirals, grids such as calendars, or along a curve

And you can combine these representations with different time scales: you could use an absolute or relative chronological scale, a logarithmic scale, or a purely sequential scale.

And then there's "Layout", or how to draw one or more timelines within a page or display:

This has to do with deciding between a single timeline, faceting into multiple timelines (like how I did for each creative person),

Or wrapping a single timeline into meaningful segments of time, like a decade or a century.

Timelines Revisited: The Research Process

1. Collecting and categorizing 145 timelines and timeline tools to establish the dimensions.
· Sources included: Cartographies of Time (Rosenberg & Grafton), Visualization of Time-Oriented Data (Aigner et al.), Making Timelines (Groeger).

2. Validating the dimensions of the design space with 118 additional timelines (263 total).
· Sources included: visual.ly, the Kantar Information is Beautiful Awards showcase, massvis.mit.edu.

3. Implementing points in the design space with 28 representative datasets.
· e.g., Conflicts, epidemics, lifespans, head of state tenures, news stories, natural disasters, publication records, geological history.

So how did I come up with this design space?

You may recall from my background section that I incorporate various forms of qualitative analysis in my work,

And that I strive to connect research and practice,

This manifested here in my collection and categorization of 145 timelines and timeline visualization tools produced by both researchers and practitioners,

And out of that came the dimensions of the design space.

Next, I verified that the design space could be used to label 118 additional timelines drawn from collections of practitioner work.

And finally I used implementation as a research method, in that I implemented points in the design space with 28 event datasets which varied in a number of ways,

Such as the number of events, the temporal extent of the data, the rate of event co-occurrence, and their topical content, which included

Biographical data, news stories, natural disasters, publication records, and extremely long-spanning timelines of geological and astronomical history.

A set of purposeful, interpretable, & generalizable
timeline designs at timelinesrevisited.github.io .

Thinking Systematically About Tasks & Design Choices

Using our Timeline Design Space

Expressive Storytelling with Timelines

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

timelinesrevisited.github.io

Provide alternative representations for time.

Provide alternative time scales.

Anticipate chronological or non-chronological narratives.

Incrementally reveal visual elements, selectively highlighting and annotating to direct attention.

In this TVCG paper I make the case that timeline design tools should provide alternative representations of time and alternative time scales.

I should also emphasize that existing timeline tools also tend towards a chronological narrative:

Either the whole timeline is shown as a static image, so readers are likely to begin at the start of the timeline, or events are revealed in chronological order, one event at a time as a series of slides.

For some stories, a chronological introduction of events makes total sense, while for others it does not,

Like how in my story about the routines of creative people, I revealed and highlighted aspects of those timelines in an order that wasn't chronological starting at midnight.

To achieve this expressive narrative design, you should make use of animation, highlighting, and annotation to incrementally reveal parts of a narrative (that may not be chronological), to allow the viewer to make new comparisons.

The Authoring Interface of Timeline Storyteller

Web version imports CSV, JSON, GSheet. Power BI version imports various data formats.
Web version exports PNG, SVG, GIF, JSON spec. Power BI version exports PBIX, iFrame.

Evaluating Timeline Storyteller

A controlled laboratory study to assess expressivity seemed to be inappropriate.

How do people use it with their own data?

How does the content they produce reflect our timeline design space?

Promoting Timeline Storyteller to Practitioners

Demos / talks at the Tapestry Conference, OpenVisConf, and the Dublin Data Summit in 2017.

Demo by Microsoft's Data Journalism Team at the 2017 Future of Storytelling Summit.

Posts on the official Power BI Blog, tutorial + interview for the Power BI YouTube channel.

Demo by a customer during the opening keynote of the 2017 Data Insights Summit:

Now in order to study how people use a deployed tool, you need to promote it to practitioners,

So I demonstrated Timeline Storyteller at visualization practitioner conferences like Tapestry and OpenVisConf.

My colleagues also contributed to this effort to attract users with their own demonstrations and presentations, including demos by Microsoft's Data Journalism Team at the Future of Storytelling Summit in 2017.

I blogged about it on the official Power BI blog,

And along with the Microsoft Data Journalism team, I co-produced an interview and tutorial video for the Power BI YouTube channel.

And finally Power BI featured a demo of Timeline Storyteller in the opening keynote of the 2017 Data Insights Summit, a major Microsoft customer conference.

This was in front of about three thousand people and recorded online, and for this demo, Power BI asked a customer to demo it using data from his organization,

Which in this case was the UK National Trust, a heritage foundation that maintains historic landmarks

And they had an interesting dataset about the chronology of several of these landmarks.

Timeline Storyteller: Collecting Usage Data

Exported content from the web version in mid 2017.

Entries from a Storytelling Contest with the Power BI user community in late 2017,
coordinated by the Microsoft Data Journalism Team.

Download metrics of the Power BI desktop version:
Over 36,000 downloads of the Power BI version as of January 2019.

Following the promotional effort, I set about collecting data on how people were using Timeline Storyteller.

Which included collecting the content that people exported from the tool, as well as collecting entries from a storytelling contest within the Power BI community coordinated by Microsoft's Data Journalism Team,

And finally we collected usage statistics from the Power BI version.

Since Power BI is a desktop product and has different protocols for data collection and retention than Microsoft Research,

I don't know as much about how people use it unless they contact us directly or post content publicly online.

However, I do know that by the end of 2018, the Timeline Storyteller component for Power BI had been downloaded over 36,000 times.

Timeline Storyteller: Content Analysis

223 unique items of exported content from the web version (subject to author consent).

The corpus spanned the timeline design space - with a couple of exceptions.

The Linear representation and Chronological time scale were most common.

Linear

I saved copies of content exported from the web version of Timeline Storyteller over the course of about 9 months, subject to the approval of content authors.

After reviewing all of this content and discarding duplicate versions of content, I identified 223 unique artefacts generated with Timeline Storyteller, which included static images, animated GIFs, and multi-scene stories.

And by qualitatively analyzing this content, I found was that nearly every timeline design choice was represented in this corpus, though most of this corpus made use of the familiar linear representation and chronological time scale.

I didn't see any use of the calendar grid or the logarithmic time scale.

And if you do a Google image search for Timeline Storyteller and look at the content produced by people other than myself,

You'll see the prevalence of these design choices reflected there.

Timeline Storyteller: Content Analysis (cont.)

Example entries from the Power BI user community data storytelling contest:

Tropical Cyclones by Manga Solutions. | TV Network Ratings by Pragmatic Works.

Timeline Storyteller: Conclusions & Opportunities

No prior interactive tools for presenting expressive timeline narratives.

The first to incorporate multi-scene stories with multiple visual representation choices.

Incrementally reveal + transform; selectively highlight + annotate; applicable to other data types.

Recommend design choices and annotations based on properties of the dataset.

In summary, Timeline Storyteller fills a gap in that no existing tool allows a non-programmer to present a visually expressive timeline narrative.

It's really the first to incorporate this animated multi-scene PowerPoint-like story format along with multiple visual representation choices;

And I believe that many aspects of its design can be applied to other forms of data, such as its ability to incrementally reveal and selectively highlight content.

We're still learning about how it's being used in the wild,

As our limited evaluation of content analysis and usage only revealed so much.

And while the content produced does reflect our timeline design space to an extent,

Seeing this content made me realize the opportunity for providing overt recommendations of design choices, captions, and annotations more overtly within the application based on properties of the dataset, such as the temporal extent and distribution of the events.

Outline

Other Expressive Information Design Tools (1 of 3)

ChartAccent: Annotation for Data-Driven Storytelling.
Ren, Brehmer, Lee, Höllerer, and Choe. In Proc. 2017 IEEE PacificVis Symp.

chartaccent.github.io | github.com/chartaccent

Other Expressive Information Design Tools (2 of 3)

Charticulator: Interactive Construction of Bespoke Chart Layouts.
Ren, Lee, and Brehmer. In IEEE TVCG (InfoVis 2018).

Honorable Mention for Best Paper at IEEE InfoVis 2018.

Shortlisted for the 2018 Kantar Information is Beautiful Awards.

charticulator.com | github.com/Microsoft/Charticulator

Other Expressive Information Design Tools (3 of 3)

DataToon: Drawing Dynamic Network Comics With Pen + Touch Interaction.
Kim, Henry Riche, Bach, Xu, Brehmer, Hinckley, Pahud, Xia, McGuffin, and Pfister. In Proc. CHI 2019.

aka.ms/DataToon

Evaluating Expressive Information Design Tools

Reflecting on the Evaluation of Visualization Authoring Systems.
Ren, Lee, Brehmer, and Henry Riche.
In Proc. BELIV 2018 (Evaluation and Beyond - Methodological Approaches for Visualization).

Lessons from evaluating tools: Timeline Storyteller, ChartAccent, & Charticulator.

Emphasis on post-deployment content analysis and chart reproduction studies.

aka.ms/renbeliv18

Beyond Tools: Evaluating Data-Driven Stories

Evaluating Data-Driven Stories & Storytelling Tools.
Amini*, Brehmer* (equal contribution), Bolduan, Elmer, and Wiederkehr.
In Data-Driven Storytelling (CRC Press 2018).

A summary of perspectives, criteria, methods, metrics, and constraints w.r.t. evaluation.

The process of looking at the content produced with Timeline Storyteller and other tools also makes me wonder about

How the visualization community should evaluate data-driven stories more generally.

Consider a story that makes use of visualization, annotation, text, and potentially interaction.

It might incorporate many design choices, in that the author was highly expressive; but was it any good?

So in book chapter that I wrote for last year's Data-Driven Storytelling book,

My co-authors and I attempted to catalog all the ways you might evaluate this form of content.

And we also did our best to consider different perspectives and constraints from both research and practice,

Which was possible as one of our co-authors was Christina Elmer, a data journalist from Der Spiegl in Germany,

And another was Benjamin Wiederkehr, who leads the Interactive Things information design firm in Switzerland.

Outline

Expressive Information Design with Pen + Touch Input

Images L to R: narrative diagram by Kurt Vonnegut, "board game"-like timeline by Matthew Lee, timeline by Mark Twain, Vonnegut drawing a narrative, a curve timeline in Timeline Storyteller.

Mobile, Responsive, & Expressive Information Design

Data Visualization on Mobile Devices. CHI 2018 Workshop organized by:
Lee, Brehmer, Choe, Isenberg, Langer, and Dachselt.

Visualizing Ranges over Time on Mobile Phones.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG (InfoVis 2018).

A Comparative Evaluation of Animation and Small Multiples
For Trend Visualization on Mobile Phones.
Brehmer, Lee, Isenberg, and Choe. Working paper (Feb. 2019).

Opportunities for Expressive Information Design

Future Research Directions

Opportunities for Expressive Information Design (1/2)

Designing and evaluating inviting and memorable techniques for presenting information.

Widening the scope of data types: e.g., spatiotemporal data, dynamic networks.

Collecting and assessing design choices from the research and practice communities.

Opportunities for Expressive Information Design (2/2)

Measuring audience graphicacy* (visual / data / statistical literacy) and identifying ways to boost it.

Information design for an audience with a limited attention span.

Mobile-first and mobile-only information design (and addressing the scarcity of research). * Uncertainty, graphicacy, and the power of statistics.
Alberto Cairo (Oct 18, 2017), thefunctionalart.com .

Another opportunity for future research pertains to studying audience constraints,

Such as their limited "graphicacy", and by this I mean a combination of data, visual, and statistical literacy.

I want to identify how to measure this construct as well as how to improve it.

There's also the constraint of the audience's limited attention span and the challenge of designing information that both attracts this attention and yet does not mislead or misinform.

Finally there's the constraint imposed by small displays, which motivates me to identify where the limitations are for visual representation and interaction design choices.

There's not a lot of prior research from the visualization community when it comes to mobile devices, and I hope to rectify this.

Considerations for Interactive & Expressive Information Design Tools

Matthew Brehmer · EPIC Research Group, Microsoft Research · @mattbrehmer

mattbrehmer.github.io#talks | slides

Presentation at Autodesk Research 2019-02-26

Supplemental

Information Design Choices on Mobile Phones

Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG (InfoVis 2018).

aka.ms/ranges-tvcg

Information Design Choices on Mobile Phones (Cont.)

A Comparative Evaluation of Animation and Small Multiples
For Trend Visualization on Mobile Phones.
Brehmer, Lee, Isenberg, and Choe. Working paper, Feb. 2019.

aka.ms/multiples | (mobile only) experimental app.

Expressive Info. Design for Mobile Devices: Ongoing

Discoverable Interactions for Navigating & Selecting Time-Varying Data on Mobile Phones.
Brehmer, Lee, Collins, and Hinckley.

Motivation: few people interact with interactive news graphics beyond scrolling.

Many interactions anticipate a desktop context, but most of the audience is using a mobile device.

UI elements for navigating multidimensional time-varying data occupy too much screen real estate.

Ranges Over Time on Mobile Phones

Detailed Experimental Design and Results Slides