Constraints and Opportunities for Expressive Information Design

Matthew Brehmer · @mattbrehmer
Presentation at Tableau Research 2019-04-09 · slides: mattbrehmer.github.io/talks/tableau190409

Outline

· My background and methods

· Considerations for expressive information design

· Timeline Storyteller

· Information design choices on mobile devices

· Opportunities for future research

My Background

2016 - 2019: Postdoctoral Researcher specializing in InfoVis + HCI, Microsoft Research

2011 - 2016: PhD Computer Science specializing in Information Visualization, UBC

2009 - 2011: MSc Computer Science specializing in Human-Computer Interaction, UBC

2004 - 2009: Bachelor of Computing specializing in Cognitive Science, Queen's Univ.

Scope of My Visualization Research

Considerations & tools for expressive information design:
· Timeline Storyteller ^{- C+J 19} | DataToon ^{- CHI 19} | Charticulator ^{- TVCG 19} | ChartAccent ^{- PVIS 17} | Timelines Revisited ^{- TVCG 17} | TimeLineCurator ^{- TVCG 16}

Visualizing information on mobile devices:
· Ranges Over Time ^{- TVCG 19} | Animation vs. Small Multiples ^{- in review} | Novel Interaction For Mobile Visualization ^{- in preparation}

Evaluating visualization (tools):
· Data-Driven Stories ^{- DDS 18} | Visualization Authoring Systems ^{- BELIV 18} | Variants of Multi-Series Bar Charts ^{- CHI 18} |
· Overview: A Document Mining Tool for Journalists ^{- TVCG 14} | Pre-Design Empiricism for Visualization ^{- BELIV 14}

Visualization task analysis:
· A Typology of Abstract Visualization Tasks ^{- TVCG 13} | Visualizing Dimensionally-Reduced Data ^{- BELIV 14}

Visualizing resource consumption:
· Workflows for Energy Portfolio Analysis ^{- TVCG 16}

Scope of My Visualization Research

Design & Research Methods

Design & Implementation:

· User interface design | Visualization design & development | Toolkit development

Qualitative Research:

· Visualization design studies | Requirements analysis | Retrospective interviews
· Chauffeured demos | Content analysis | Post-deployment usage analysis

Quantitative Research:

· Laboratory experiments | Crowdsourced experiments | Statistical analysis

In terms of how I do my research,

I spend a lot of time designing and building things

I've alternated between visualization and interface design at various levels of fidelity,

as well as software development for a deployed product, a research toolkit, and various visualization tools and experimental applications.

Though just as important are the human-centred research methods that I use,

This includes quite a bit of qualitative evaluation work during my PhD in the context of visualization design or case studies,

Which involved everything from requirements analysis to analyzing the usage of deployed visualization tools.

This qualitative work also involves systematically collecting and categorized previous research as well as information design content done by practitioners.

And finally I also do lab-based experimental work, and especially recently with some experiments involving crowdsourcing,

I'll revisit some of these methods later in this talk when I speak about evaluating interactive visualization tools.

Enabling Information Design

How can I enable under-served groups of people to...

Expressively visualize their data?

Produce and present compelling data-driven stories?

Applications of visualization beyond those in professional data analysis.

With respect to the values and goals that underlie my research,

I'm really hoping to enable more people to visualize and present compelling stories about their data,

And its in this respect that I contrast my research from that of many others in the broader visualization community, such as those working in visual analytics.

Many in the visualization and visual analytics research community are motivated by use cases from professional data analysis, data science, or business intelligence.

While I think these use cases are important and I'm glad that people are thinking about them, they are NOT the use cases that motivate my research.

Instead, I'm interested in incorporating visualization to present information to people who don't identify as data analysts, in both public and personal contexts.

So I often think of news readers, students, policy makers, and citizens who consume visual displays of information and visual explanations.

The Value of Connecting Research & Practice

Disseminating visualization research into practice, and vice versa.

Promoting and studying the adoption of deployed information design tools and research prototypes.

Collecting examples of information design produced by practitioners.

Fostering a dialogue between researchers and practitioners (e.g., OpenVisConf, VisInPractice)*. * OpenVisConf: openvisconf.com | VisInPractice event at IEEE VIS: VisInPractice.github.io .

Finally, increasing expressivity and democratizing visualization are only possible if there is an ongoing process of communication and outreach between visualization researchers and practitioners.

This means disseminating research into practice, and just as importantly, disseminating practice into research.

As you'll see in my own research, this manifests as studying the adoption of deployed tools and prototypes.

And collecting examples of information design produced by practitioners.

And finally this means speaking directly with practitioners, such as when I spoke at OpenVisConf, one of the most renowned practitioner-oriented visualization conferences,

Or how I co-organized the VisInPractice event at last year's IEEE VIS conference, where I invited prominent visualization practitioners to speak to a largely academic audience.

Expressive Information Design

From the perspective of an information visualization researcher.

Expressive Information Design

· Combining visualization, annotation, and explanation to present information to an audience.

· Thinking systematically about tasks, design choices, and constraints.

· Identifying ways to assess alternative design choices.

Presenting Information to the Public

e.g., Hans Rosling's TED presentations about global economic and public health indicators.
Image: Open Knowledge Foundation Deutschland (flickr, cc by).

Presenting Information to the Individual

Image: newkemall (flickr, cc by).

Aspects of Expressive Information Design

Thinking systematically about tasks, design choices, and constraints.

Thinking Systematically about Tasks

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

With respect to tasks, a central theme of my PhD research pertained to classifying visualization tasks and to consolidating the vocabulary for visualization tasks into a typology,

Which is the subject of a 2013 paper written with my PhD advisor Tamara Munzner.

And expressive information design requires being able to address a wide range of tasks,

Including both the tasks of an information designer or author, as well as the tasks of a viewer, or what the audience is expected to do with the information presented to them.

Such as to identify a trend or to compare values.

Ultimately, expressivity is about satisfying the intent of the author, allowing them to tell the story they want to tell,

And whether this story can be reconstructed in the mind of the viewer.

Thinking Systematically about Tasks

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

The most cited IEEE InfoVis paper since 2013, with more than 300 citations*.

* Google Scholar, Apr 2019.

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

Thinking Systematically about Design Choices

Identifying the dimensions of design spaces that characterize:

... ways to visually represent data.

... ways to animate and interact with these representations.

... ways to highlight and annotate them.

... ways to combine visual content with textual explanation.

In an expressive information design tool, how do you present these choices?

e.g., Low-level visual encoding choices (shape, color, ...) or chart templates (bar, line, ...)

Expressivity also refers to thinking systematically about design choices and design spaces:

Which are ways to categorize visual representation choices for a particular type of data,

Interaction design choices, ways to highlight, annotate, and embellish visual representations,

Or ways to combine visual and textual explanations in compelling and engaging ways.

Characterizing these design spaces becomes particularly important when you decide to expose them to an author in the form of an information design tool, where there is a question about the level of granularity:

For example, as a designer are you controlling low-level visual encoding channels like shape or color, or are you picking a chart type from a template?

As you'll see an example of this process of identifying and assessing points in a design space in the next section of this talk .

Constraints on Expressivity

Constraints imposed by authors, by the audience, or by the context.

Expertise: empowering non-programmers and non-designers.

Resources: enabling expressive information design for those on a deadline.

Literacy: anticipating the visual and data literacy of the audience.

Device: anticipating the audience's viewing experience.

But first I want to mention that expressivity is also bound to constraints, and it is these constraints that I find to be particularly challenging and deserving of research.

And these include constraints imposed by designers or authors, as well as those imposed by the audience and the context in which the information is presented.

<< CLICK >> One of these constraints is expertise:

I want to empower non-programmers to present information with interactive visualization and presentation tools.

And I want to empower non-designers to make compelling and useful information graphics by offering design guidance.

<< CLICK >> Another constraint is about resources:

For example, I want to enable over-committed, understaffed, and poorly funded local news teams to produce data-driven stories about their community while meeting all their deadlines.

<< CLICK >> Yet another constraint is literacy:

What if my audience lacks the sufficient visual literacy or data literacy needed to interpret my particular choice of visual representation?

How can I make use of their prior knowledge as part of the communication process?

<< CLICK >> And finally there's the device: what if my audience is consuming this information from a mobile device?

How does that constrain my choices?

Outline

· My background and methods

· Considerations for expressive information design

· Timeline Storyteller

· Information design choices on mobile devices

· Opportunities for future research

The Daily Routines of Famous Creative People

Story inspired by infographics by Podio and info we trust.
Data source: Daily Rituals: How Artists Work by Mason Currey (2013)

To illustrate this, I want to tell you a few interesting things about the daily routines of famous creative people.

MAX: What you see here is a set of radial timelines depicting a typical 24 hours in the lives of 26 writers, artists, composers, and other creative types.

When they work, eat, sleep, exercise, and do other activities.

NEXT: I had wondered if there was any pattern as to when creative people do their creative work.

It turns out that at least among these people, some creative people prefer to work in short, focused bursts, like Darwin or Kant, while people like Murakami and Voltaire get up early and work for long uninterrupted periods of time.

Meanwhile, you have odd ones like Kafka who work all night long.

NEXT: I had also wondered about the potential relationship between sleep and creativity, so let me focus your attention to these blue arcs.

Most people here tend to sleep at night, with notable exceptions like Balzac who would turn in at 6pm, and Kafka, who just seemed to keep odd hours.

A third question I had was about variation and creativity. Let me show you how often these creative people change activities throughout their day.

NEXT: This 24-hour clock metaphoe isn't the best way to convey the number or heterogeneity of activities, so I'm going to transition to a just show you a sequence of activities without any duration, effectively resulting in a radial bar chart.

NEXT: To show who varied the most and least among these people, a linear representation is probably better, where you can see that Darwin has the most varied day of all the people here while Murakami prefers less variation.

NEXT: Finally, I'll restore the duration of events so that you can compare these timelines just by scanning up and down, where you might spot times where some routines appear to be in sync with one another, such as when people work or sleep at the same time.

And this is a good time to ask you to compare your own daily rhythm to these creative people.

When do you sleep, eat, or do you best creative work?

Expressive Information Design with Timelines

Timeline Storyteller: The Design & Deployment of an Interactive Authoring Tool for
Expressive Timeline Narratives.
Brehmer, Lee, Henry Riche, Tittsworth, Lytvynets, Edge, and White. In Proc. Comp. + Journalism 2019.

timelinestoryteller.com | github.com/Microsoft/timelinestoryteller

Timelines Revisited

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

Timelines are visual representations of categorical event sequences.

How have people drawn timelines over the course of history?

The visualization research community has focused on their use in data analysis.

How have practitioners used them for storytelling?

What Happened When?

In what sequence did the events occur?

How long did the events last?

How long between event A and event B?

Did A and B co-occur or repeat?

When did A and B occur relative to event C?

A Timeline Design Space

Scale

Layout
Unified

It turns out that there are many ways to answer these questions and present timelines, which brings me to my design space.

You can represent time in various ways, such as linearly, radially, or by using spirals, grids such as calendars, or along a curve

And you can combine these representations with different time scales: you could use an absolute or relative chronological scale, a logarithmic scale, or a purely sequential scale.

And then there's "Layout", or how to draw one or more timelines within a page or display:

This has to do with deciding between a single timeline, faceting into multiple timelines (like how I did for each creative person),

Or wrapping a single timeline into meaningful segments of time, like a decade or a century.

Timelines Revisited: The Research Process

1. Collecting and categorizing 145 timelines and timeline tools to establish the dimensions.
· Sources included: Cartographies of Time (Rosenberg & Grafton), Visualization of Time-Oriented Data (Aigner et al.), Making Timelines (Groeger).

2. Validating the dimensions of the design space with 118 additional timelines (263 total).
· Sources included: visual.ly, the Kantar Information is Beautiful Awards showcase, massvis.mit.edu.

3. Implementing points in the design space with 28 representative datasets.
· e.g., Conflicts, epidemics, lifespans, head of state tenures, news stories, natural disasters, publication records, geological history.

So how did I come up with this design space?

I mentioned earlier that I incorporate various forms of qualitative analysis in my work,

And that I strive to connect research and practice,

This manifested here in my collection and categorization of 145 timelines and timeline visualization tools produced by both researchers and practitioners,

And out of that came the dimensions of the design space.

Next, I verified that the design space could be used to label 118 additional timelines drawn from collections of practitioner work.

And finally I used implementation as a research method, in that I implemented points in the design space with 28 event datasets which varied in a number of ways,

Such as the number of events, the temporal extent of the data, the rate of event co-occurrence, and their topical content, which included

Biographical data, news stories, natural disasters, publication records, and extremely long-spanning timelines of geological and astronomical history.

A set of purposeful, interpretable, & generalizable
timeline designs at timelinesrevisited.github.io .

Thinking Systematically About Tasks & Design Choices

Using our Timeline Design Space

Expressive Storytelling with Timelines

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

timelinesrevisited.github.io

Provide alternative representations for time.

Provide alternative time scales.

Anticipate chronological or non-chronological narratives.

Incrementally reveal visual elements, selectively highlighting and annotating to direct attention.

In this TVCG paper I make the case that timeline design tools should provide alternative representations of time and alternative time scales.

I should also emphasize that existing timeline tools also tend towards a chronological narrative:

Either the whole timeline is shown as a static image, so readers are likely to begin at the start of the timeline, or events are revealed in chronological order, one event at a time as a series of slides.

For some stories, a chronological introduction of events makes total sense, while for others it does not,

Like how in my story about the routines of creative people, I revealed and highlighted aspects of those timelines in an order that wasn't chronological starting at midnight.

To achieve this expressive narrative design, you should make use of animation, highlighting, and annotation to incrementally reveal parts of a narrative (that may not be chronological), to allow the viewer to make new comparisons.

The Authoring Interface of Timeline Storyteller

Web version imports CSV, JSON, GSheet. Power BI version imports various data formats.
Web version exports PNG, SVG, GIF, JSON spec. Power BI version exports PBIX, iFrame.

Evaluating Timeline Storyteller

A controlled laboratory study to assess expressivity seemed to be inappropriate.

How do people use it with their own data?

How does the content they produce reflect our timeline design space?

Promoting Timeline Storyteller to Practitioners

Demos / talks at the Tapestry Conference, OpenVisConf, and the Dublin Data Summit in 2017.

Demo by Microsoft's Data Journalism Team at the 2017 Future of Storytelling Summit.

Posts on the official Power BI Blog, tutorial + interview for the Power BI YouTube channel.

Demo by a customer during the opening keynote of the 2017 Data Insights Summit:

Now in order to study how people use a deployed tool, you need to promote it to practitioners,

So I demonstrated Timeline Storyteller at visualization practitioner conferences like Tapestry and OpenVisConf.

My colleagues also contributed to this effort to attract users with their own demonstrations and presentations, including demos by Microsoft's Data Journalism Team at the Future of Storytelling Summit in 2017.

I blogged about it on the official Power BI blog,

And along with the Microsoft Data Journalism team, I co-produced an interview and tutorial video for the Power BI YouTube channel.

And finally Power BI featured a demo of Timeline Storyteller in the opening keynote of the 2017 Data Insights Summit, a major Microsoft customer conference.

This was in front of about three thousand people and recorded online, and for this demo, Power BI asked a customer to demo it using data from his organization,

Which in this case was the UK National Trust, a heritage foundation that maintains historic landmarks

And they had an interesting dataset about the chronology of several of these landmarks.

Timeline Storyteller: Collecting Usage Data

Exported content from the web version in mid 2017.

Entries from a Storytelling Contest with the Power BI user community in late 2017,
coordinated by the Microsoft Data Journalism Team.

Download metrics of the Power BI desktop version:
Over 36,000 downloads of the Power BI version as of January 2019.

Following the promotional effort, I set about collecting data on how people were using Timeline Storyteller.

Which included collecting the content that people exported from the tool, as well as collecting entries from a storytelling contest within the Power BI community coordinated by Microsoft's Data Journalism Team,

And finally we collected usage statistics from the Power BI version.

Since Power BI is a desktop product and has different protocols for data collection and retention than Microsoft Research,

I don't know as much about how people use it unless they contact us directly or post content publicly online.

However, I do know that by the end of 2018, the Timeline Storyteller component for Power BI had been downloaded over 36,000 times.

Timeline Storyteller: Content Analysis

223 unique items of exported content from the web version (subject to author consent).

The corpus spanned the timeline design space - with a couple of exceptions.

The Linear representation and Chronological time scale were most common.

Linear

I saved copies of content exported from the web version of Timeline Storyteller over the course of about 9 months, subject to the approval of content authors.

After reviewing all of this content and discarding duplicate versions of content, I identified 223 unique artefacts generated with Timeline Storyteller, which included static images, animated GIFs, and multi-scene stories.

And by qualitatively analyzing this content, I found was that nearly every timeline design choice was represented in this corpus, though most of this corpus made use of the familiar linear representation and chronological time scale.

I didn't see any use of the calendar grid or the logarithmic time scale.

And if you do a Google image search for Timeline Storyteller and look at the content produced by people other than myself,

You'll see the prevalence of these design choices reflected there.

Timeline Storyteller: Content Analysis (cont.)

Example entries from the Power BI user community data storytelling contest:

Tropical Cyclones by Manga Solutions. | TV Network Ratings by Pragmatic Works.

Timeline Storyteller: Conclusions & Opportunities

No prior interactive tools for presenting expressive timeline narratives.

The first to incorporate multi-scene stories with multiple visual representation choices.

Incrementally reveal + transform; selectively highlight + annotate; applicable to other data types.

Recommend design choices and annotations based on properties of the dataset.

In summary, Timeline Storyteller fills a gap in that no existing tool allows a non-programmer to present a visually expressive timeline narrative.

It's really the first to incorporate this animated multi-scene PowerPoint-like story format along with multiple visual representation choices;

And I believe that many aspects of its design can be applied to other forms of data, such as its ability to incrementally reveal and selectively highlight content.

We're still learning about how it's being used in the wild,

As our limited evaluation of content analysis and usage only revealed so much.

And while the content produced does reflect our timeline design space to an extent,

Seeing this content made me realize the opportunity for providing overt recommendations of design choices, captions, and annotations more overtly within the application based on properties of the dataset, such as the temporal extent and distribution of the events.

Outline

· My background and methods

· Considerations for expressive information design

· Timeline Storyteller

· Information design choices on mobile devices

· Opportunities for future research

Information Design Choices on Mobile Phones

Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG (InfoVis 2018).

aka.ms/ranges-tvcg

Image: Eric Boam's 7 Months of Sleep.

Image: The New York Times (June 4, 2017).

Images (clockwise from left): The New York Times · Eric Boam · Randy Olson.

Ranges in Weather Apps

Apps: Dark Sky © · Weathertron © · Weather Line ©.

Ranges in Sleep Tracking Apps

Apps: Azumio Sleep Time © · Garmin Connect © · Bedtime (iOS Clock) ©.

Ranges and Radial Representations

Image: Weather Radials (by Timm Kekeritz) ©.

Images: mobileinfovis.com by Sebastian Sadowski, 2018.

A Crowdsourced Experiment on Mobile Phones

The first crowdsourced visualization evaluation study performed exclusively on phones. Image: newkemall (flickr, cc by).

Thinking Systematically About Tasks

Tasks derived from A Multi-Level Typology of Abstract Visualization Tasks.
Brehmer and Munzner. In IEEE TVCG (InfoVis 2013):

· Locate Dates

· Identify Values

· Locate Extreme Values

· Compare Values

· Compare Ranges

Dependent Measures

For each trial:

Trial completion time

Response accuracy

At each level of granularity:

Preference: Linear or Radial

Confidence: Low to High

Participants

Temperature (N = 40), Sleep (N = 47)

84 trials per participant, using their own phone.

Radial or Linear?

Detailed statistics are provided in:
Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG (InfoVis 2018). aka.ms/ranges-tvcg .

People are, in general, slower with radial representations.

Accuracy appears to be data- and task-dependent:

e.g., less accurate with radial when identifying and locating values in absence of seasonal variation.

People prefer and are more confident with linear representations.

In the interest of time, I won't get into the detailed stats including confidence intervals and effect sizes for each task, which you can find in the paper; for now, I'll stick to high-level results.

In general, people tend to be slower with radial representations, irrespective of the source of the data.

However, radial representations don't seem to incur accuracy costs, at least with temperature range data, where there tends to be a lot of seasonal variation.

With sleep range data, which exhibits no seasonal variation, people are less accurate when identifying values or when locating extreme values.

It's also worth remarking that participants universally preferred linear representations, and they felt more confident using linear representations relative to radial ones.

Altogether these findings are interesting given the popularity of radial representations of time in practice and design communities.

Week vs. Month vs. Year

People are typically slower with a month than with a week of ranges.

For some tasks, people were less accurate with a month than with a year.

e.g., seasonal variation in annual temperature appears to be beneficial for locating extreme values.

As you might expect, people were slower with a month ranges than with a week of ranges,

but they weren't that much slower when they moved from a month to a year.

With regards to accuracy, the jump from a week to a month only incurred noticeably worse performance when locating extreme values.

Interestingly, there were cases where people were as accurate or even more accurate with a year of ranges than with a month of ranges, particularly among those looking for extreme temperatures.

It is in this respect that I observed different results between the temperature and sleep range participant groups,

Where it would seem as though the expected seasonal trend of temperature ranges appears to make the task easier relative to examining a month of sleep ranges.

Temperature (L) and Sleep (R) don't follow monthly cycles.

Ranges Over Time on Mobile Phones: Conclusions

Is a cycle meaningful in the context of the data?

Does the task involve locating values? Or comparing them?

Is efficiency important?

Locating values quickly? → Choose Linear.

Comparing values (and unconcerned with speed)? → Choose Radial or Linear.

Ranges Over Time on Mobile Phones: Opportunities

More research is needed to assess visualization design choices on mobile phones.

aka.ms/ranges | (mobile only) experimental app.

github.com/Microsoft/RangesOnMobile | open source app and analysis.

medium.com/multiple-views-visualization-research-explained | blog post for practitioners.

Expressive Info. Design for Mobile Devices, Continued

A Comparative Evaluation of Animation and Small Multiples
For Trend Visualization on Mobile Phones.
Brehmer, Lee, Isenberg, and Choe. In review, Apr. 2019.

aka.ms/multiples | (mobile only) experimental app.

Outline

· My background and methods

· Considerations for expressive information design

· Timeline Storyteller

· Information design choices on mobile devices

· Opportunities for future research

Opportunities for Expressive Information Design

Designing and evaluating inviting and memorable techniques for presenting information.

Widening the scope of data types: e.g., spatiotemporal data, dynamic networks.

Collecting and assessing design choices from the research and practice communities.

Opportunities for Expressive Information Design (cont.)

Measuring audience graphicacy* (visual / data / statistical literacy) and identifying ways to boost it.

Information design for an audience with a limited attention span. * Uncertainty, graphicacy, and the power of statistics.
Alberto Cairo (Oct 18, 2017), thefunctionalart.com .

Opportunities for Expressive Information Design (cont.)

New input modalities for expressive information design, annotation, and presentation.

e.g., DataToon: Drawing Dynamic Network Comics With Pen + Touch Interaction.
Kim, Henry Riche, Bach, Xu, Brehmer, Hinckley, Pahud, Xia, McGuffin, and Pfister. In Proc. CHI 2019.

Opportunities for Expressive Information Design (cont.)

Mobile-first and mobile-only information design (and addressing the scarcity of research).

Responding to the demand for mobile business intelligence.

Constraints and Opportunities for Expressive Information Design

Matthew Brehmer · @mattbrehmer

mattbrehmer.github.io/talks/tableau1904019 | slides Presentation at Tableau Research 2019-04-09