Visualizing Quantities & Events Over Time

From Analysis to Communication, For Displays Both Large & Small

Matthew Brehmer · InfoVis + HCI Researcher · mattbrehmer.github.io
Presentation at Chatham Labs 2019-04-25 · slides: mattbrehmer.github.io/talks/chatham190425

Outline

· My background, methods, & inspiration

· A time series visualization client project

· Expressive information design with timelines

· Visualizing time-oriented data on mobile devices

· Current research interests

My Background

2016 - 2019: Postdoctoral Researcher specializing in InfoVis + HCI, Microsoft Research

2011 - 2016: PhD Computer Science specializing in Information Visualization, UBC

2009 - 2011: MSc Computer Science specializing in Human-Computer Interaction, UBC

2004 - 2009: Bachelor of Computing specializing in Cognitive Science, Queen's Univ.

Scope of My PhD & Postdoc Research

Visualization task analysis:
· A Typology of Abstract Visualization Tasks ^{- TVCG 13} | Visualizing Dimensionally-Reduced Data ^{- BELIV 14}

Evaluating visualization (tools):
· Data-Driven Stories ^{- DDS 18} | Visualization Authoring Systems ^{- BELIV 18} | Variants of Multi-Series Bar Charts ^{- CHI 18} |
· Overview: A Document Mining Tool for Journalists ^{- TVCG 14} | Pre-Design Empiricism for Visualization ^{- BELIV 14}

Time series visualization:
· Workflows for Energy Portfolio Analysis ^{- TVCG 16}

Considerations & tools for expressive information design:
· Timeline Storyteller ^{- C+J 19} | DataToon ^{- CHI 19} | Charticulator ^{- TVCG 19} | ChartAccent ^{- PVIS 17} | Timelines Revisited ^{- TVCG 17} | TimeLineCurator ^{- TVCG 16}

Visualizing time-oriented data on mobile devices:
· Ranges Over Time ^{- TVCG 19} | Animation vs. Small Multiples ^{- in review} | Novel Interactions with Time-Oriented Data on Mobile Devices ^{- in preparation}

Scope of My PhD & Postdoc Research

Design & Research Methods

Design & Implementation:

· User interface design | Visualization design & development

Qualitative Research:

· Visualization design studies (client projects) | Requirements analysis | Retrospective interviews
· Chauffeured demos | Think-aloud protocols | Post-deployment usage analysis

Quantitative Research:

· Laboratory experiments | Crowdsourced experiments | Statistical analysis

In terms of how I do my research,

I spend a lot of time designing and building things

I've alternated between visualization and interface design at various levels of fidelity,

as well as software development for a deployed visualization tool, experimental applications, and interactive sandbox visualization design environments.

Though just as important are the human-centred research methods that I use,

This includes quite a bit of qualitative evaluation work during my PhD in the context of visualization design studies,

which is the visualization community's way of saying "client projects",

Which tend to involve everything form requirements analysis to analyzing the usage of deployed visualization tools using various methods,

And finally I also do lab-based experimental work, and especially recently with some experiments involving crowdsourcing,

Visualization Task Analysis

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

Visualization Task Analysis

A Multi-Level Typology of Abstract Visualization Tasks. Brehmer and Munzner.
In IEEE Transactions on Visualization and Computer Graphics (InfoVis 2013).

The most cited IEEE InfoVis paper since 2013, with more than 300 citations*.

* Google Scholar, Apr 2019.

Icons by Eamonn Maguire (cc by) for Munzner's Visualization Analysis & Design (CRC Press, 2014).

Goal: Enabling Information Visualization

How can I enable people to visualize their data in context-appropriate ways?

From public presentation via large displays to individual consumption via mobile displays.

Applications of visualization beyond those in professional data analysis.

With respect to the values and goals that underlie my research,

I'm hope to enable more people to visualize their data in compelling and contextually-appropriate ways.

And its in this respect that I contrast my research from that of many others in the broader visualization community, such as those working in visual analytics.

Many in the visualization and visual analytics research community are motivated by use cases from professional data analysis, data science, or business intelligence.

While I think these use cases are important and I'm glad that people are thinking about them, they are NOT the use cases that motivate my recent research.

Instead, over the past couple of years I've been interested in incorporating visualization to present information to people who don't identify as data analysts, in both public and personal contexts.

So I often think of news readers, students, policy makers, and citizens who consume visualization.

Inspiration: Connecting Research & Practice

Disseminating visualization research into practice, and vice versa.

Promoting and studying the adoption of deployed visualization tools and research prototypes.

Collecting examples of visualization design by practitioners (e.g., news graphics designers).

Fostering a dialogue between researchers and practitioners (e.g., OpenVisConf, VisInPractice)*. * OpenVisConf: openvisconf.com | VisInPractice event at IEEE VIS: VisInPractice.github.io .

With respect to where my inspiration for research comes from, I often look to the visualization practitioner community.

While disseminating research into practice is an important goal, I think that it is just as important to inform research with an understanding of current practices.

In my own research, this manifests as studying the adoption of deployed visualization tools including my own work.

And collecting examples of visualization design produced by practitioners.

I also prioritize speaking directly with practitioners, such as when I spoke at OpenVisConf in 2017, one of the most renowned practitioner-oriented visualization conferences,

Or how I co-organize the VisInPractice event at the IEEE VIS conference, where I invite visualization practitioners to speak to a largely academic audience about their craft.

Outline

· My background, methods, & inspiration

· A time series visualization client project ^{- TVCG 16}

· Expressive information design with timelines

· Visualizing time-oriented data on mobile devices

· Current research interests

Energy Usage in Building Portfolios

Matches, Mismatches, and Methods: Multiple-View Workflows for Energy Portfolio Analysis.
Brehmer, Ng, Tate, and Munzner. In IEEE TVCG 2016 (InfoVis 2015).

Given a portfolio of buildings (e.g., a university campus, a hotel / restaurant chain):

· Summarize energy usage patterns across the portfolio,

· Drill Down from the portfolio to a subset of buildings,

· Identify anomalies and buildings in need of energy-saving interventions,

· Compare buildings to one another, or before / after events & interventions,

Reduce energy consumption and save money.

This was a project that I presented at InfoVis 2015

Done in collaboration with an energy analytics company in Vancouver

Whose clients were facilities managers of large building portfolios,

Such as university campuses, school boards, or hotel / restaurant chains,

Where the buildings have the capacity to continuously report their energy use.

They ultimately wanted to conserve energy and save money

And in order to do so, they needed to summarize energy use across their portfolio,

Drill down to subsets of buildings,

Identify anomalies and opportunities to save energy,

l As well as compare buildings to one another and to themselves over time,

And particularly after the implementation of any energy-saving intervention.

Our Partner Company's Existing Tool

Little support for summarizing energy usage across a portfolio (no overview).

Clients did not trust / understand predictions and could not easily make comparisons.

Clients were using tool's export feature to do their own analysis in Excel.

Evaluating Alternative Design Choices

Many design choices considered (e.g., faceted boxplots, bar + bump plots, stacked area charts, faceted line graphs, calendar tile matrix, ...).

Established conventions in energy management limited the use of bar charts and line graphs.

However, clients were unfamiliar with alternative design choices.

After learning about the domain and the clients' tasks,

I set about designing alternative ways to summarize the energy usage of a building portfolio,

Ways that would allow for clients to make better comparisons between buildings and over time.

These are some of the designs I considered.

Once notable constraint I faced was that bar charts and line charts, two of the most familiar representations,

Have a strong domain convention associated with them, in that line charts were used exclusively for raw energy demand and

Bar charts were used for consumption,

And as a result it would be confusing to use these representations for other derived or aggregate quantity that you would show in a summary.

Chauffeured Sandbox Demos with Clients

Built a web-based sandbox visualization tool (incl. aggregation, filtering, & normalization).

Conducted chauffeured demos with clients with their own portfolio data.

Identified multiple-view workflows (juxtaposition, sequencing) that addressed clients' tasks.

Results: Realization in Partner's Software

Based on my research with clients, the partner company incorporated my designs into their tool.

Juxtaposing unfamiliar choices (tile matrix and boxplots) superior to either in isolation.

Drill-down from tile matrix to faceted bar charts and line graphs respected domain conventions.

I extensively documented the results of these client sessions and the recommended workflow designs,

And as a result our collaborator eventually incorporated some of my designs into the next version of their software.

These designs involved juxtaposing some previously unfamiliar representations,

Specifically a calendar tile matrix and an interactive boxplot;

Two designs that hadn't been well-received in isolation,

But by juxtaposing and interactively linking them, they reinforced one another.

This combination served well as an overview, which allowed clients to drill down to faceted line and bar charts,

Depending on their unit of interest, be it energy demand or consumption,

and this design also respected their domain conventions.

Results: Realization in Partner's Software

Matches, Mismatches, and Methods: Multiple-View Workflows for Energy Portfolio Analysis.
Brehmer, Ng, Tate, and Munzner. In IEEE TVCG 2016 (InfoVis 2015).

Segue: From Time Series to Timelines

· From PhD to postdoctoral research,

And...

· From quantities to events over time,

· From analysis to communication.

Outline

· My background, methods, & inspiration

· A time series visualization client project

· Expressive information design with timelines ^{- TVCG 17, C+J 19}

· Visualizing time-oriented data on mobile devices

· Current research interests

The Daily Routines of Famous Creative People

Story inspired by infographics by Podio and info we trust (RJ Andrews).
Data source: Daily Rituals: How Artists Work by Mason Currey (2013)

To give you an example of a way to present timelines, I want to show you a brief presentation about the daily routines of famous creative people.

MAX: What you see here is a set of radial timelines depicting a typical 24 hours in the lives of 26 writers, artists, composers, and other creative types.

When they work, eat, sleep, exercise, and do other activities.

NEXT: I had wondered if there was any pattern as to when creative people do their creative work.

It turns out that at least among these people, some creative people prefer to work in short, focused bursts, like Darwin or Kant, while people like Murakami and Voltaire get up early and work for long uninterrupted periods of time.

Meanwhile, you have odd ones like Kafka who work all night long.

NEXT: I had also wondered about the potential relationship between sleep and creativity, so let me focus your attention to these blue arcs.

Most people here tend to sleep at night, with notable exceptions like Balzac who would turn in at 6pm, and Kafka, who just seemed to keep odd hours.

A third question I had was about variation and creativity. Let me show you how often these creative people change activities throughout their day.

NEXT: This 24-hour clock metaphor isn't the best way to convey the number or heterogeneity of activities, so I'm going to transition to a just show you a sequence of activities without any duration, effectively resulting in a radial bar chart.

NEXT: To show who varied the most and least among these people, a linear representation is probably better, where you can see that Darwin has the most varied day of all the people here while Murakami prefers less variation.

NEXT: Finally, I'll restore the duration of events so that you can compare these timelines just by scanning up and down, where you might spot times where some routines appear to be in sync with one another, such as when people work or sleep at the same time.

And this is a good time to ask you to compare your own daily rhythm to these creative people.

When do you sleep, eat, or do you best creative work?

Expressive Information Design with Timelines

timelinestoryteller.com | github.com/Microsoft/timelinestoryteller

Timelines Revisited

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

Timelines are visual representations of categorical event sequences.

How have people drawn timelines over the course of history?

The visualization research community has focused on their use in data analysis.

How have practitioners used them for storytelling?

... A survey of 250+ timelines and timeline tools.

In late 2014, I became fascinated by the potential for expressive information design with timelines,

Which are used to tell stories in domains such as journalism, education, and project management.

So I collected and categorized how people have drawn timelines over the course of history.

I did so upon realizing that prior visualization research involving timelines had focused on their uses in data analysis, such as for understanding patterns in electronic health records.

And because of this focus, the expressive range of design choices considered by the research community was far smaller than the range of design choices considered by practitioners using timelines to communicate.

And eventually I had collected over 250 timeline and timeline visualization tools which led to a design space for presenting timelines.

A Timeline Design Space

Scale

Layout
Unified

It turns out that there are many ways to present timelines.

You can represent time in various ways: linearly, radially, or by using spirals, grids such as calendars, or along a curve

And you can combine these representations with different time scales: you could use an absolute or relative chronological scale, a logarithmic scale for extremely long-spanning timelines, or you could use a purely sequential scale.

And then there's "Layout", or how to draw one or more timelines within a page or display:

This has to do with deciding between a single timeline, faceting into multiple timelines (like how I did for each creative person),

Or wrapping a single timeline into meaningful segments of time, like a decade or a century.

Considerations for Storytelling with Timelines

Timelines Revisited: A Design Space and Considerations for Expressive Storytelling.
Brehmer, Lee, Bach, Henry Riche, and Munzner. In IEEE TVCG (presented at InfoVis 2017).

timelinesrevisited.github.io

Different combinations of representation, time scale, and layout serve different communicative tasks.

Anticipate chronological or non-chronological narratives.

Incrementally reveal visual elements, selectively highlighting and annotating to direct attention.

These considerations led to the development of Timeline Storyteller.

In this TVCG paper I make the case that timeline design tools should provide alternative representations of time, time scales, and layouts,

As different combinations of these serve different communicative tasks.

For instance, sometimes I want to emphasize sequence without duration, other times periodicity.

I also discuss in this paper how existing timeline tools also tend towards a chronological narrative:

Either the whole timeline is shown as a static image, so readers are likely to begin at the start of the timeline, or events are revealed in chronological order, one event at a time as a series of slides.

For some stories, a chronological introduction of events makes total sense, while for others it does not,

Like how in my story about the routines of creative people, I revealed and highlighted aspects of those timelines in an order that wasn't chronological starting at midnight.

To achieve this expressive narrative design, you should make use of animation, highlighting, and annotation to incrementally reveal parts of a narrative (that may not be chronological), to allow the viewer to make new comparisons.

Each of these considerations were realized in the tool Timeline Storyteller.

The Authoring Interface of Timeline Storyteller

Web version imports CSV, JSON, GSheet. Power BI version imports various data formats.
Web version exports PNG, SVG, GIF, JSON spec. Power BI version exports PBIX, iFrame.

The Impact of Timeline Storyteller

Demo by a customer during the opening keynote of the 2017 Data Insights Summit

Demos / talks at Tapestry, OpenVisConf, the Dublin Data Summit, and the Future of Storytelling.

A storytelling contest with the Power BI community (via the Microsoft Data Journalism Team).

Over 36,000 downloads of the Power BI version as of end of 2018.

We launched Timeline Storyteller in mid 2017, and since then it has had some impact in the visualization community.

Our big success came when Microsoft Power BI featured a demo of Timeline Storyteller in the opening keynote of the 2017 Data Insights Summit, a major Microsoft customer conference.

This was in front of about three thousand people and recorded online, and for this demo, Power BI asked a customer to demo it using data from his organization,

Which in this case was the UK National Trust, a heritage foundation that maintains historic landmarks,

And they had an interesting dataset about the chronology of several of these landmarks.

I also demonstrated Timeline Storyteller at visualization practitioner conferences like Tapestry and OpenVisConf.

My colleagues also contributed to this effort to attract users with their own demonstrations and presentations, including demos by Microsoft's Data Journalism Team at the Future of Storytelling Summit in 2017.

Since Power BI is a desktop application and we don't collect telemetry about its usage, we don't know much about how people have used Timeline Storyteller unless they contacted us or posted content online.

But we do know that by the end of 2018, it had been downloaded over 36,000 times.

Timeline Storyteller: Conclusions & Opportunities

Timeline Storyteller: The Design & Deployment of an Interactive Authoring Tool for
Expressive Timeline Narratives.
Brehmer, Lee, Henry Riche, Tittsworth, Lytvynets, Edge, and White. In Proc. Comp. + Journalism 2019.

No prior interactive tools for presenting expressive timeline narratives.

The first to incorporate multi-scene stories with multiple visual representation choices.

Incrementally reveal + transform; selectively highlight + annotate; applicable to other data types.

Future Work: Recommend design choices and annotations based on properties of the dataset.

In summary, Timeline Storyteller fills a gap in that no existing tool allows a non-programmer to present a visually expressive timeline narrative.

It's really the first to incorporate this animated multi-scene PowerPoint-like story format along with multiple visual representation choices;

And I believe that many aspects of its design can be applied to other forms of data, such as its ability to incrementally reveal and selectively highlight content.

Though we have collected only a few hundred examples of content that people produced using Timeline Storyteller that they elected to share with us,

Seeing this content made me realize the opportunity for providing overt recommendations of design choices, captions, and annotations more overtly within the application based on properties of the dataset, such as the temporal extent and distribution of the events.

Outline

Experiment 1: Ranges over Time on Mobile Phones

Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG 2019 (InfoVis 2018).

aka.ms/ranges-tvcg

Ranges in Sleep Tracking Apps

Apps: Azumio Sleep Time © · Garmin Connect © · Bedtime (iOS Clock) ©.

Ranges in Weather Apps

Apps: Dark Sky © · Weathertron © · Weather Line ©.

Ranges and Radial Representations

Image: Weather Radials (by Timm Kekeritz) ©.

Images: mobileinfovis.com by Sebastian Sadowski, 2018.

A Crowdsourced Experiment on Mobile Phones

The first crowdsourced visualization evaluation study performed exclusively on phones. Image: newkemall (flickr, cc by).

Thinking Systematically About Tasks

Tasks derived from:

A Multi-Level Typology of Abstract Visualization Tasks.
Brehmer and Munzner. In IEEE TVCG (InfoVis 2013):

· Locate Dates

· Identify Values

· Locate Extreme Values

· Compare Values

· Compare Ranges

Dependent Measures

For each trial:

Trial completion time

Response accuracy

At each level of granularity:

Preference: Linear or Radial

Confidence: Low to High

Participants

Temperature (N = 40), Sleep (N = 47)

84 trials per participant, using their own phone.

Radial or Linear?

Detailed statistics are provided in:
Visualizing Ranges over Time on Mobile Phones: A Task-Based Crowdsourced Evaluation.
Brehmer, Lee, Isenberg, and Choe. In IEEE TVCG 2019 (InfoVis 2018).

aka.ms/ranges-tvcg

People are, in general, slower with radial representations.

Accuracy appears to be data- and task-dependent:

e.g., less accurate with radial when identifying and locating values in absence of seasonal variation.

People prefer and are more confident with linear representations.

In the interest of time, I won't get into the detailed stats including confidence intervals and effect sizes for each task, which you can find in the paper; for now, I'll stick to high-level results.

In general, people tend to be slower with radial representations, irrespective of the source of the data.

However, radial representations don't appear to incur accuracy costs, at least with temperature range data, where there tends to be a lot of seasonal variation.

With sleep range data, which exhibits no seasonal variation, people are less accurate when identifying values or when locating extreme values.

It's also worth remarking that participants universally preferred linear representations, and they felt more confident using linear representations relative to radial ones.

Altogether these findings are interesting given the popularity of radial representations of time in practice and design communities.

Week vs. Month vs. Year

People are typically slower with a month than with a week of ranges.

For some tasks, people were less accurate with a month than with a year.

e.g., seasonal variation in annual temperature appears to be beneficial for locating extreme values.

Temperature (L) and Sleep (R) don't follow monthly cycles.

Ranges Over Time on Mobile Phones: Conclusions

Is a cycle meaningful in the context of the data?

Does the task involve locating values? Or comparing them?

Is efficiency important?

Locating values quickly? → Choose Linear.

Comparing values (and unconcerned with speed)? → Choose Radial or Linear.

aka.ms/ranges | (mobile only) experimental app.

github.com/Microsoft/RangesOnMobile | open source app and analysis.

As for conclusions, the questions of which representation and which granularity to display are really questions about congruence between these design choices and the combination of data and task.

Designers should ask whether a cycle is meaningful in the context of the data,

Whether the task is about locating values or comparing them,

And whether task efficiency is the first priority.

My hope for this work was to inspire the visualization research community to consider the future of visualization design choices on mobile devices as well as the potential research questions that can be asked via crowdsourced experimentation,

And to this end I open-sourced my experimental application and the data analysis code so that others could run similar experiments on mobile phones,

though substituting different data and chart types.

Experiment 2: Animation vs. Small Multiples

A Comparative Eval. of Animation & Small Multiples for Trend Visualization on Mobile Phones.
M. Brehmer, B. Lee, P. Isenberg, and E. K. Choe. (InfoVis 2019 Submission).

Highlights: Animation or Small Multiples?

Results question the "small multiples on the desktop, animation on the phone" design pattern.

Surprisingly, small multiples are still viable on small displays.

They may be particularly well suited for distance and angle comparisons.

Animation and small multiples are comparably accurate in many cases.

Animation may be preferable for cases of outlier trajectories and reversals.

aka.ms/multiples | (mobile only) experimental app.

github.com/Microsoft/MobileTrendVis | open source app and analysis.

Outline

Opportunities for Expressive Information Design

New input modalities for expressive information design, annotation, and presentation.

Pen + Touch Interaction for Authoring Data Comics

DataToon: Drawing Dynamic Network Comics With Pen + Touch Interaction.
Kim, Henry Riche, Bach, Xu, Brehmer, Hinckley, Pahud, Xia, McGuffin, and Pfister. In Proc. CHI 2019.

aka.ms/DataToon

Expressive Timeline Design with Pen + Touch Input

Images L to R: narrative diagram by Kurt Vonnegut, "board game"-like timeline by Matthew Lee, timeline by Mark Twain, Vonnegut drawing a narrative, a curve timeline in Timeline Storyteller.

Goal: Enabling Information Visualization

Visualizing Quantities & Events Over Time

From Analysis to Communication, For Displays Both Large & Small

Matthew Brehmer · InfoVis + HCI Researcher · mattbrehmer.github.io

mattbrehmer.github.io/talks/chatham190425 | slides Presentation at Chatham Labs 2019-04-25