We investigate direct manipulation of graphical encodings as a method for interacting with visualizations. There is an increasing interest in developing visualization tools that enable users to perform operations by directly manipulating graphical encodings rather than external widgets such as checkboxes and sliders. Designers of such tools must decide which direct manipulation operations should be supported, and identify how each operation can be invoked. However, we lack empirical guidelines for how people convey their intended operations using direct manipulation of graphical encodings. We address this issue by conducting a qualitative study that examines how participants perform 15 operations using direct manipulation of standard graphical encodings. From this study, we 1) identify a list of strategies people employ to perform each operation, 2) observe commonalities in strategies across operations, and 3) derive implications to help designers leverage direct manipulation of graphical encoding as a method for user interaction.
We developed an interactive data visualization using JavaScript. The visualization modules are built using the D3 toolkit. The tool we implemented has an Upload and a Reset button. The Upload button enables the interviewer to upload any of the visualizations created using the datasets used in our study. The Reset button allows the interviewer to reset the visualization after a participant has interacted with the visualization. We added interactivity to the encodings used in visualizations. For instance, users could change the position, color, width, and height of the bars in a bar chart. Or, they could change the size, position, and color of circles in the scatterplot. The tool shows one visualization at a time. Participants can perform each operation by directly manipulating the encodings in the visualization. For example, the interviewer uploads a bar chart visualization in our tool and asks a participant to perform an operation. The participant can manipulate different operations to convey her intention. The tool only enables participants to manipulate the encodings and does not recompute the visualization (similar to a drawing interface like Adobe Illustrator). Enabling the participants to convey their actions without the system reacting to those actions allowed us to observe participants' unrevised behavior, and drive system design to accommodate it.
In total participants employed 203 strategies to perform 15 different operations. We first sketched out each participant's strategy and grouped these sketches into 48 independent categories based on their similarities. Each sketch belongs only to one of the identified categories and all categories are exclusive. We then independently named each category of strategies (e.g.,``Stack points vertically'') and counted the total number of sketches ineach category. The interactive table below shows in rows the 15 operations participants performed during the study, and in columns the 48 strategies we identified. Each cell shows the number of times participants used the strategy in column to perform the operation in row. The higher the value in a cell, the darker the background of the cell. Hovering on each strategy shows its description.
Names | Reposition a few Points Rearrange a few points to sort them based on their values | Stack Points Vertically Rearrange a few points to position them vertically where they have some overlaps | Reposition Points Vertically Rearrange a few points to position them vertically where they have no overlap | Regroup a few Points Rearrange a few points to put them close together | Reposition to Target Move a point to the target value | Retop to Target Moving a bar so the top of the bar aligns with the target value | Reposition to Tick Mark Move a point to the tick mark representing the target value | Overlay Bars Move one bar on the top of another bar so they overlay | Sloping Bars Move bars vertically to sort them based on their heights to create ascending or descending slope | Reposition Tallest & Shortest Move the tallest bar to one extreme, then the shortest bar to another extreme | Stack two Bars Stack one bar on the top of another one | Reposition Bars by Height Move a few bars based on their heights | Resize a few Points Equally Resize a few points equally | Resize & Reposition to Axis Resize a few points, then move them to the tick marks on the axis | Resize Point Resize a point to the target value | Select & Resize Select a set of points and resize them | Resize by Values Resize a few data points differently based on their values | Cover & Resize Make one point big to cover few other points | Recolor one Group Color few points that fall in the same range using the same color | Recolor Groups Color few points that fall in different ranges differently | Select & Recolor One Select all points, then color one of them | Select & Recolor Group Select a set of points and color them | Recolor by Values Color a set of points based on their values | Recolor a few Points Color a few points using the same color | Reposition & Resize Rearrange a few points to sort them based on their values, then resize them | Stack & Enclose Rearrange a few points to stack them vertically, then resize a circle to enclose the stack | Reposition to Target & Resize Reposition a point to target value, then resize it | Resize & Move out Resize a point, then move it out of the visualization | Reposition Inline and Recolor Rearrange a few points vertically/horizontally, then color them | Reposition by Range & Recolor Rearrange a few points based on a range of values, then color them | Recolor, Reposition & Recolor Color a point, then move a second point to the target value and color it the same color as the first point | Reposition to Target & Recolor Move the point to the target value, then color it | Regroup & Recolor Move a few points closer together, then color them | Recolor & Paint Change the color of one data point, then move it over other points | Recolor & Dip Color a point, then move other data points over it | Resize a Point and Recolor Resize a single point, then color it | Resize & Recolor Resize points based on their values, then color them | Cover & Recolor Make one point big to cover other points, then color it | Reheight Ascending Changing the height of the bar in an ascending order | Reheight to Target Changing the height of the bar to the target vlaue | Enwidth Increase the width of the bar | Enheight & Enwidth Increase the height of the bar to exceed the x axis, then increase its width | Reposition & Enwidth Drag two bars close together, then increase the width of one of the bars | Recolor & Enwidth Color two bars and increase the width of one of the bars to cover all the other bars | Enwidth & Recolor Increase the width of the bar, then color it | Enwidth & Cover Increase the width of a bar to cover all the other bars, then color it | Intersect & Recolor Drag the bottom of the bar to intersect with the tick mark, then color it | Recolor & Reheight Color the bar, then change its height to the target value |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
O1: Assign a data attribute to an axis | 7 | 3 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||
O2: Switch from a scatterplot to a bar chart | 2 | 5 | 1 | 3 | 2 | 1 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||
O3: Navigate the point over time in a scatterplot | 5 | 4 | 5 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||
O4: Adjust the value of a data point | 3 | 1 | 2 | 4 | ||||||||||||||||||||||||||||||||||||||||||||
O5: Group the two bars into one bar | 9 | 2 | 3 | 1 | ||||||||||||||||||||||||||||||||||||||||||||
O6: Sort the bar chart | 1 | 3 | 9 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||
O7: Change the size of all data points | 8 | 4 | 1 | 1 | ||||||||||||||||||||||||||||||||||||||||||||
O8: Assign a data attribute to the size of points | 1 | 1 | 7 | 2 | ||||||||||||||||||||||||||||||||||||||||||||
O9: Change the color of all points | 3 | 5 | 1 | 2 | 1 | 2 | ||||||||||||||||||||||||||||||||||||||||||
O10: Assign a data attribute to the color of points | 1 | 8 | 1 | |||||||||||||||||||||||||||||||||||||||||||||
O11: Change the color of all bars | 3 | 7 | 2 | 2 | 2 | |||||||||||||||||||||||||||||||||||||||||||
O12: Assign a data attribute to the color of all bars | 10 | 1 | 1 | |||||||||||||||||||||||||||||||||||||||||||||
O13: Navigate the point over time in a bar chart | 4 | 9 | 1 | |||||||||||||||||||||||||||||||||||||||||||||
O14: Adjust the value of a bar | 2 | 9 | 2 | |||||||||||||||||||||||||||||||||||||||||||||
O15: Expand the range of a bin in a histogram | 3 | 1 | 9 | 1 | 1 |