Michael Friendly recently sent me this Common Sense Revolution visualization by Scott Sørli plotting a timeseries from 1985-2007 of welfare income for a single person in Ontario; and the names of all the homeless who died on the streets of Toronto over the same time period. An inverse correlation is strongly apparent implying a potential causal relation between the welfare amount and the homeless deaths. While the deaths could have been a simple line chart or bar chart, stacked names much more strongly indicate that we’re dealing with people. And more so that a stack of people icons, these are named people: real people with real given names, real surnames and presumably families and connections in their communities, such as Floyd Anderson, Cheryl Lynn Gunn or Norma/n Lewis. And, disappointingly, there are quite a few John Does and Jane Does, where presumably the investigators did not have enough resources to track down the real names of the deceased homeless person.
It’s also a reminder that text visualizations have a long history. In my book, I do look at a lot of historical text visualizations – as a basis for creating a framework for considering the many ways data can be encoded into text. And then given the framework, I create many visualizations.
But it’s also highly useful and relevant to continue to look at historic examples, to see techniques, combinations, and methods that may inspire or inform future visualizations and creative works. I recently found a copy of Language & Structure in North America (November 4-30, 1975, Richard Kostelanetz curator). Here’s a few interesting snaps of visualization-like uses of text from the 1970’s:
Leftmost is portion of George Maciunas‘ The history of Fluxus, a text-centric flow chart organized by time indicating in historical art movements leading up to Fluxus. The polar plot is an analytical diagram by Agnes Denes titled Studies of Time/ Exploration of Time Aspects, plotting concepts vs time past/present/future further organized by dimensions such as memory – a prioi knowledge, and reproductive – modification. Noise Text #1 by Ascher/Straus is a result of a series of transformations on texts into what appears to be a set of textual vectors.
Visualizing prosody isn’t new. Here’s a great example from 1969 by Ernest/Marion Robson, using letter width to indicate duration, and font-weight to indicate intensity as well as a baseline shift. Not surprisingly, the encoding is very similar to the example visualizations which I’d created as these are connotative mappings. I like their much more dramatic variation in width and use of all caps, overplotting, and use of leaders (….) and whitespace (from Introduction to Transwhichics, DuFour Editions, PA, 1969):
And here’s a very interesting creation of a 3D visualization based on an analysis of syllables per unit measure from Yeats by Beth Learn 1975 (Timeslide Over/Time):
The final two examples are generative works, creating new text from pre-existing work. On the left, a receipt is used as the basis for constraining words by Karen Shaw titled $8.40 (1975) (did not find a good link for Karen). Each line item on the receipt sets the cost per word, where each letter has a unique cost. Words are then stacked into two alternative poems:
On the right, John Perreault, Goddess, 1969, uses parentheses to mark words within larger words or spanning across words, e.g. “(Eve)n in(to t)h(in)e own (so)ft-(con)che(d ear):” thereby creating alternative readings.
Creating and understanding alternative texts becomes more important with an increase in computational textual analytics. Whether overlaying analyses such as attention or assessing generative text sequences, these artistic approaches hint at some possibilities for visualizing text.
The tragic events in Ukraine have left me wondering how quantitative visualizations miss showing complex issues such as human rights. One aspect of this conflict mentioned by various media outlets as well as elected officials is the flow of funds to purchase commodities, particularly oil, helps fund the military ambitions of the state. While Russia’s human rights record is terrible, many other oil-exporting nations also have serious human rights issues. How might difficult concepts such as political risk and human rights be shown in a visualization about oil?
In visualization, a quick solution would be to find a metric which encodes risk, rights and freedoms. A metric is needed because: a. Visualizations encode quantitative (and categorical) data, not unmeasured data; b. You can’t manage what you can’t measure.
These are commonly-held wisdoms in visualization and management consulting. But is this the right approach? Consider a treemap of oil exports from countries (showing only countries with more than 100,000 barrels per day):
The primary encoding of the treemap is oil exports by size. Saudi Arabia is the largest, but also Russia, Iran, Iraq, UAE, Kuwait, Nigeria and Canada are large as well – each exporting more than 1.5m barrels per day. At $100/barrel, that’s more than $150m/day. The dollar amounts are enormous, creating enormous opportunities for sovereign governments to use some portion of that money for state activities.
Not all countries are bad actors. Color in this treemap indicates political risk, as indicated by a risk rating. However, this particular risk rating doesn’t rate some countries such as Norway and Mexico – presumably the level of risk is not similar between these countries.
Thus, we might look a metric with better coverage. The treemap below uses the Corruption Perception Index (from Transparency International) for color:
In this example there is coverage across all countries. Russia, Iran, Iraq and many others look bad, Libya, South Sudan and Venezuela worse (although this data has not been updated in response to the invasion of Ukraine). The color scale is a diverging scale, copied from a map on the Wikipedia article indicating Corruption Perception Index. Unfortunately, this creates green for countries implying good scores – including for some countries with poor human rights records.
Therefore, we might try to keep searching for a metric (and a color scale), that better captures what we think should this metric should show. This search for metrics is an attempt to capture our real-world knowledge of risks and rights abuses of different countries, but we’re also in danger of simply looking for metrics that confirm our biases. Here’s a nicer version of the treemap perhaps a bit closer to our expectations using the Global Peace Index and the inferno color scale:
All of these indexes attempt to capture complex multi-variate data. For example, an American viewer may object the the Peace Index categorizing United States at the same level as Algeria. If no single metric captures these issues, one might turn to a visualization technique that instead shows many variables, such as parallel coordinates. But creating a much more complex visualization, misses the simple immediacy of the treemap – and ignores that all these size-based visualizations (bar charts, pie charts, treemaps, sunbursts, area charts, etc) are highly prevalent and will continue to be popular.
What to do?
Annotations in areas
Many visualizations use size to draw attention to larger objects: bar charts, pie charts, maps, treemaps, etc. In all the treemaps above, Saudi Arabia and Russia are large, Gabon and Vietnam are not. Presumably, the largest exporters should have more scrutiny, not just a larger size.
Interestingly in cartographic maps – such as a roadmap, Google map, etc – large areas end up with more labels. Why shouldn’t visualizations do the same? After all, the largest areas are the items with much larger values, and thus perhaps deserve more attention than the tiny items. Here’s the treemap visualization again, this time with the opening paragraph or lede sentences from Human Rights Watch country pages:
In this example, the treemap remains and the color coding remains. Large blocks also have additional text that can be directly read if of interest. Saudi Arabia’s human rights record indicate issues with official accountability for the murder of Jamal Khashoggi; Russia’s record indicates it is the most repressive since the Soviet era (and this is text from before the attack on Ukraine); UAE detains dissidents even after completing their sentences (and UAE is positively biased on both the peace index and corruption index). Even large exporting countries with generally good records, such as Canada and USA, now have enough space to indicate rights issues such as the rights of Indigenous peoples in Canada, or poverty and inequality in USA.
The different kinds of rights issues not visible with a singular metric have the opportunity to become directly visible with the addition of annotations. There is space to shine a light on the details behind the largest exporters. Income inequality and Indigenous issues are human rights issues as are other repressions, but the viewer can make a more informed comparison about the instances, breadth, severity and cruelty of the largest exporters. Abstract concepts such as peace and corruption are made more concrete with instances and examples.
This example helps to turn the concept of a generic commodity (oil) into a more uncomfortable question about where the money goes after you pay to fill up your vehicle, turn on your stove, or take another flight.
Many visualization people are against dual-axis charts. There are research papers that recommend against, websites that recommend against, and some tools don’t support it. Hadley Wickham, the author of the popular R visualization package ggplot2, does not support dual axes charts. Critics point out that they can be misunderstood and the viewer may attend to the wrong visual elements, such as line crossings or relative position between lines.
Instead, consider this informative dual-axis chart showing cigarette sales and lung cancer mortality from Our World in Data:
The chart clearly shows: – Two data sets: cigarette sales and lung cancer deaths – Rising and falling trends in each are highly visible – Labels and color-coding clearly distinguish the data, lines, axes and tick labels. – Annotations indicate key events A viewer can, at a glance, see that the shape of lung cancer deaths roughly mirrors the shape of cigarette sales, with roughly a 20 year lag.
Critics don’t like multiple y-axis charts for many reasons. However, in this chart, many of these problems have been addressed. Here’s a few issues with dual axis charts: – Axes can be confused, but that is less likely here due to color-coding and titles at the top of each axis. – Line crossings are visually salient, but this chart does not draw attention to the line crossing, instead annotations draw attention to other events. – Comparisons can be gamed, for example, by tweaking the start and end timeframe and the relative scales of the axes, once can manipulate where crossings occur or the slopes of lines. Here, a shared zero baseline and similar peak heights indicate that the chart isn’t being gamed.
Detractors might also suggest a derived chart could be used, such as a rolling correlation between the values — however, that assumes the viewer understands a rolling correlation, and further the base data is lost (if not shown), or a more complicated set of cross-references between charts is required. This chart provides the base data, and allows the viewer to make perceptual comparisons between the series.
I’ve actively created visualization tools with multiple y-axes – in use by hundreds of thousands of users(!). And I’ve written a research paper (Y2Y) on dual axis charts (together with Eugene Sorenson and Craig Hagerman). Here’s some more evidence and reasoning in support of cases for multiple y-axes charts.
A. Financial Services Software
In financial services software, multiple y-axes charts are common. In Refinitiv’s Eikon, the Bloomberg Terminal, or Cosaic timeseries charts, the financial professional can create charts not only with 2 y-axes, but more:
B. Financial Services Publications
In financial services publications, multiple y-axes charts are common. We looked at 25 publications by financial firms, such as banks, mutual funds, advisory firms and central banks (pg 14-16 PDF). Across 925 pages there were 1305 charts, of which 944 were timeseries charts. Out of those 944 timeseries charts, 179 where dual axes charts. That is: 19% of the timeseries charts use dual axes. Here’s a few examples:
Clearly financial professionals are comfortable with the use of dual axis charts in their communications, and not concerned with mis-interpretation of these charts. You can also find examples in financial news, such as The Economist and the Financial Times:
C. Example Use Case for a Dual Y-Axis Chart
C. We provided a few specific examples. Here is an example of price comparison between two financial securities – the price of oil, and the price of the Canadian dollar, to show how the dual axis chart aids the analytical user. First, here’s a chart of each, side by side:
Both start low, go up, then drop back down, even lower than their starting low. This is expected, because Canada produces and exports a lot of oil. But the price of the Canadian dollar isn’t directly linked to the price of oil, and it doesn’t always follow the price of oil either – notice the sharp drop in the price of oil in 2014 whereas the Canadian dollar has a long decline for 2012 to 2015. Questions such as “which series started rising first” cannot be determined by looking at these charts. How about putting the charts together, with a single axis:
A single axis does not work for comparison, as the Canadian dollar is valued in fractions of a US dollar whereas barrels of oil trade around $40-100 US dollars. Instead, these two series could be normalized to a starting value of 100, and here is the resulting chart with a single normalized axis:
A normalized axis is a bit more useful. In absolute terms, over the 8 year period, the price of oil more than doubles, while the Canadian dollar increases perhaps 30%. Price of oil is more volatile than the Canadian dollar.
It’s still hard to see patterns in the dollar. Both price lines zigzag, but which one leads over the other? Do they always move in the same direction? These are critical questions to commodities traders and currency traders. This information impacts whether they can correctly assess price movement, and whether or not they make or lose money. Here’s the two charts plotted vertically, with aligned time axis:
The alignment helps to see that many of the local dips and local peaks share trends, but it’s still hard to see when they might be off by a day or two, or if they always move in tandem. Finally, here’s a dual axis chart:
The similar shapes help make lines close to each other and this facilitates local visual comparisons. Most of the time in this chart, the series tend to move in the same direction at the same time, but periods of divergence are also visible. For example, in the larger yellow shaded zone in late 2013/early 2014 oil moves up while the Canadian dollar moves down. Similarly, after a strong rally in both in the first few months of 2016, oil stays relatively unchanged while the Canadian dollar starts a downward trend.
The example is discussed in more detail in the research paper, including examples with scatterplots, a horizon chart and rolling correlation charts. This is also further discussed with more examples of other charts; and a discussion on automating when to use dual-axis charts in a chapter in a forthcoming book by Springer later this year.
Making Effective Dual Axis Charts
In our charts as well as the cigarette and cancer chart you’ll notice a number of cues to help the viewer:
Consistent color-coding of the series (i.e. the line in the plot area), the series name (whether in the title, the top of axes, or in a legend), the tick marks, and the tick labels. Basically, if the graphical bit going onto the chart relates to only one of the series, color it to match that series. Color-coded series names at the top of the axes are helpful to the viewer so that they don’t have to glance back and forth between the axes and a separate legend.
Don’t manipulate the vertical ranges to distort the data: both series should have similar tops and similar bottoms; or possibly aligned to make the lines close for local comparison.
Keep the grid lines simple. Don’t try to draw 2 different sets of horizontal grid lines — it creates clutter. Try to align the ticks and gridlines, or if that doesn’t work, only show the axes ticks and labels and skip one or both sets of grid lines.
Most importantly, know your audience. If your audience is unfamiliar with dual-axes charts, consider alternative charts. Or, if using dual-axes charts with an unfamiliar audience, more care is required to draw attention to the meaningful insights the chart shows: such as the use of titles that indicate key insights, or annotations to specific observations that the viewer should attend to.
Visualizing with Text footnote – Snapchat text chart
I’m seeing examples of interesting text visualizations in the wild. These are relevant to my book Visualizing with Text, particularly if I find examples that don’t quite fit. Occasionally, I’ll pop an example into the blog. Today’s example is from Snap Inc.’s 2021 Annual Report. It’s a timeseries chart with the area under the line filled with various Snap projects at each time interval. Conceptually, it fits into Chapter 6: Distributions:
Over the holidays, we put out a couple jigsaw puzzles, to solve collaboratively or otherwise take a break from holiday mayhem. These are big puzzles, we did a 2000 piece cartoon puzzle and a 1000 piece photograph puzzle.
With such big puzzles, different strategies are used — for example, a color-blind family member is more reliant on texture over color. Strategies for solving a jigsaw puzzle should be interesting to visualization researchers, because many of the tasks to solve a jigsaw puzzle are visualization-like activities:
– Search: finding pieces of that match a subset of visual criteria, based on shape, color, text, texture, etc. – Locate: finding the place in the portion already solved to insert the new piece. –Identify: find a singular unique piece with unique criteria.
To do these tasks, we might use many different visual properties of the puzzle pieces:
A. Shape: The first step is to find the edge pieces and solve the perimeter. While shape is generally not considered preattentive by visualization researchers – it is for finding border pieces. Puzzle borders are straight, all other puzzle pieces are curvy or jaggy — meaning it’s visually preattentive to quickly find those straight edge pieces in a sea of curvy bits.
B. Similarity: Find pieces that share similar features. For example, in the cartoon, this included: – Text: Puzzle pieces with text on them; – Color: Pieces of a similar color (red), then given many red pieces, subdividing those into bright red (helicopter), soft red (stucco wall), red with brick texture (brick wall); – Stripes: green/chartreuse stripes(an awning); or black lines a regularly spaced intervals (pickets on handrails) — stripes are a type of texture; – Shape: tree branches (brownish branchy shapes) and leaves (a ragged zigzag on light green or dark green); – Blur: Interestingly, there is no blur in the cartoon puzzle, but int the photographic puzzle blur was a useful cue. The photo had a sharp focus at one depth with increasing blurriness at further/nearer depths. This was a useful cue for sorting pieces by depth in the scene.
Categorization: Note that this similarity task is not trying to decode identity of a specific piece, rather it’s a categorization task, similiar to unsupervised clustering. In some respects, it differs from current data science approaches to clustering where the number of clusters needs to be defined upfront. Instead, the number of clusters is never defined, and is interactively refined throughout the puzzle-solving process.
C. Alignment: Given the similar pieces, use secondary cues to align pieces, e.g.: – Slope/angle: the green/chartreuse stripes form an awning and the stripes slope at a gradually varying angle: this angle is used to locate adjacent pieces – Texture orientation: the bricks and mortar are in a regular pattern meaning pieces can be rotated to the correct orientation. Stone courses at regular intervals then help locate adjacent pieces. – Texture spacing: lines that represent a handrail are pickets. Pickets are spaced regularly. Two pieces may be adjacent if the spacing between the pickets within a piece match the spacing across the two pieces.
D. Content inspection: Near the end of the puzzle solve, the puzzle was largely solved except for highly detailed pieces without strong continuity of color/texture/shape between adjacent pieces (e.g. scenes with lots of little people). In this case content analysis was required and consideration of associations, e.g. a crowd of people shouting, a room full of many technical devices and so on.
E. Other strategies: Not all strategies are visual! One person’s strategy was a trial fit: if the color/texture is close, try to jam the piece in. If it doesn’t fit, no need to visually scrutinize the piece.
So What? Puzzle solving uses visual features such as shape, texture, texture orientation, texture pattern regularity, and blur (in addition to color). These tend to be used infrequently in data visualizations, but might have potential to be used more effectively.
Song lyrics depend heavily on rhythm, syllables and rhyme (in some songs such as pop songs). Some poetry visualizations add white space between words and lines, which can then be filled with various visualization techniques, such as forming links between related words. Instead, if a lyric is considered like a stacatto sequence of syllables, the layout is more akin to a set of tiles locked together. Then instead of whitespace, visualization is constrained to the tiles.
Simple tiles with English and phonetic syllables
To start, consider Billie Eilish’s Bad Guy. Similar sounds (e.g. rhyme) don’t visibly pop-out in English text. Our goal is to encode those to make them visible. A simple approach is to convert English words to phonetic alphabet, so that the same sounds have the same phonetic symbol:
You can visually scan the phonetic symbols, but you have to look closely at the letter shapes: Rhymes are driven by the vowel sound, which may or may not be at the end of the syllable. Furthermore, in the international alphabet, some vowel sounds are represented by a single symbol and some are represented by two symbol thus making it difficult to attend to the relevant symbols. With phonetic symbols, sounds are comparable, but don’t visually pop-out.
Color-coded vowel sounds
How to make the sounds visually pop-out? Each syllable is a collection of phonemes for vowels and consonants, typically leading consonant(s), vowel(s), and trailing consonant(s). However, there are ~23 consonant phonemes and 16 vowel phonemes in English. Encodings such as brightness, font-weight, etc., don’t scale well to 16-23 uniquely discernible categories. Color is a possibility color — particularly given that some phonemes are similar sounding. Using a confusion matrix, colors can be chosen so that close-sounding sounds have similar colors (although vowel frontness and vowel origin matrix might be better).
Here is a variation where the phoneme is split into three parts: – leading consonant in light italic serif font – central vowel in heavyweight sans font, color coded to the vowel sound, with similar sounds in similar colors – trailing consonant sounds in a heavyweight serif font
You can easily scan and notice similar vowel sounds in final syllable of each line, plus the trailing consonant – aka the rhymes (e.g. gaɪ). You might also notice some other phonetic techniques such as the leading repetition in the chorus meɪk / maɪt, or near rhymes such as ˈkrɪ–mə–nəl / ˈsɪ–nɪ–kəl.
On the otherhand, using the phonetic alphabet results in some unfamiliar symbols for most native English speakers, e.g. ʌ for “uh” or ʃ for “sh”.
Instead, the tile background can be color-coded and the text switched to English spelling:
But the sound of the trailing consonant has been lost: guy and type have the same vowel sound, but don’t perfectly rhyme due to differing trailing consonant. Worse, nose, toes, and knows, actually do rhyme but are spelled quite differently.
Fun with a polychromatic font
A polychromatic font is a font specifically designed for use with multiple colors. There are a few different fonts that support multiple colors, by providing multiple versions of the font that align overtop each other. Mostly these fonts are available for purchase, not freely available. The example below uses the font Up up and away:
In the example, below, the inside color is the vowel sound, the outside color (and the gratuitous 3D) is the final consonant sound. If there is no final consonant, then background color is used:
This is just for fun – “Hey, I’ve got this great font, let’s try it out and see what happens”. It has long been known that adjacent colors influence the perception of a color. In practice, this would never work perceptually for effective visualization but could make some viscerally-exciting data-driven text. And some of the color combinations aren’t very legible. See Josef Albers Interaction of Color for awesome paintings of the effect:
Textures! (plus color and text)
Finally, we get to a version with a tile where: – English text is used per tile – Color indicates the vowel sound – Texture indicates the final consonant sound (if no consonant, then no texture)
Since color is dominant, it can be seen the guy and type are the same color and thus the same vowel sound. However, type, with the ending p sound, gains the p texture, thus differentiating it from guy. Tough, rough, e-nough all share the same color with puffed, but the texture change gives away the slightly different color between puffed and the others.
Colors are created so that similar vowel sounds have similar colors. Likewise for textures, similar consonant sounds attempt to have similar textures. If rhyme is largely based on the vowel and trailing consonant, this color and texture per syllable create visible patterns across the tiles, visually showing rhyming scheme as well as other phonetic devices. Note similarities also at beginning of lines, e.g. Sleepin‘/ Creepin‘, or Own me/ I’ll be/ with me/ If she/ pity.
At a high-level, sub-columns of same color, same (or similar) trailing consonant visually standout revealing some of the textual structure running through sections of the lyrics.
Brig really (really) likes Abba. What happens when we use this to visualize Dancing Queen?
Many rhyming pairs are immediately apparent: scene / queen; low / go; swing / king; guy / high. And near rhymes stand out too: queen / sweet / teen / beat / rine all share the long E vowel (purple), and flip between a trailing n or t (diamond hatch vs horizontal line). The near match is also apparent in jive / life (both purple but sawtooth vs x texture).
At a more meta-level, Dancing Queen seems to have more of a blue/purple consistency compared to Bad Guy that tends to be purple and punctuated with other other distinct colors such as cyan and chartreuse.
What about something that isn’t quite so pop music, less lyric driven? Everything above is focused purely on words, i.e. poetry. Pitch, duration and the many other music variables haven’t been considered, and certainly there are many other music visualization techniques (e.g. Ethan Hine, Brian Cort). A linguistic musician tells me genres may use near rhymes rather than perfect rhymes, or may alter the inflection or pronunciation of words to get rhymes (thanks Craig). So, here’s grandson’s Dirty:
It is more difficult to define line length and color appears more random as well. There’s no predominant color across the entire lyrics. Unlike Bad Guy and Dancing Queen, there are no columns of color although there are some localized pockets of color. Perfect rhyming pairs exist, such as silence/ violence; sunset / up yet; neighbor / nature; but don’t prevail. There are some near rhymes too such as so go / to go / do you or floorboard / forewarned. There’s a lot more repetition of singular words such as time, you, love, for. And the tiles also help show near repetition of phrases such as: is it time / is it in / isn’t that; or do you love / do you have.
So perhaps the approach also works, but in this case different aspects are lyrics are creating different patterns and potentially different or additional elements need to be visualized as well.
Note: A rough implementation of the above is available as an Observable notebook. I had a few challenges with fonts and leveraged Riccardo Scalco’s texture.js to create the many different textures.
Rudolf Modley was a key figure in the popularization of Isotype in the United States. I’ve previously written about Isotype (e.g. hypothesizing what happened to it, and thematic axes). I recently received Modley and Lowenstein’s book Pictographs and Graphs(1952, Harper & Brothers). In addition to some beautiful pictographic charts, it also includes useful explanations of the design process and rationale used to create these effective and engaging charts. Here’s some insights from 70 years ago:
Insights from Modley
Storytelling. Modley was talking about storytelling with charts a half-century before data journalism: “The pictorial chartmarker is a headline writer among statisticians. If he fails to tell a story, his charts become pointless.” – pg 23.
Pictographs. “Pictorial symbols should be self-explanatory” – pg 25. A worthy goal, but a big challenge for anyone who’s had to try to design an icon for a menu (hamburger icon? gear?) or CPI (inflated $? balloon?)
Comparisons.“Pictographs make comparisons, not flat statements.” – pg 26. A single row of pictographs referring to a single value is pointless. It’s about comparing one value to another. There are quite a few infographics that fall into this category, with “one big number” and associated pictograph, but what’s it compared to? On it’s own, it’s a single factoid without any potential relative judgment.
Memorable charts.“A good chart may be judged from what the reader remembers the day after he sees it.” – pg 28. Modley sets the stage for this need right at the beginning of the book, on page 2, he describes Mr. Smith consuming information throughout the data – “a flood of varying facts which he must digest and evaluate for himself. Not the least important problem is to retain the essential facts from the wealth of information passing through his mind in one day”.
Personal engagement.“The American development of pictorial statistics has tried to avoid over-standardization of symbols. … it has wanted to bring symbols to life and to adapt them to each new audience. As we have seen in the case of Mr. Smith, his full interest and curiosity are not aroused unless there is some suggestion of his own habits and interests in a graph or illustration.” This is an interesting indication that rather than uniform pictographs used across all charts (perhaps like early Isotype before Gerd Arntz), Modley instead recognizes a requirement for icons intrinsically connected with the subject matter.
Here’s a couple examples in action from the book:
The left image shows the number of women at work – a straight-forward Isotype-like chart with the subtle cue of women’s attire changing with successive rows. This subtle change indicates, minimally that each row represents different data. Further, the attire change reinforces the time scale by using attire associated with each period.
In the right image, a person is comically attempting to hold a pile of coins. The person is literally staggering under a pile of debt (an idiom made into a visualization!). Note the captions above each column indicating the dollar amount – relative visual comparisons are possible, and the quantitative facts are explicitly depicted as well.
I understand from Nigel Holmes via Jason Forrest, that this 1952 book reprints only some of the content from Modley’s earlier book from 1937 How to Use Pictorial Statistics (a much more rare book). One day I’ll have to track down an edition.
Visualizing with Text footnote – 2 letter Scrabble words.
I’m seeing examples of interesting, interactive text visualizations in the wild. These are relevant to my book Visualizing with Text, particularly if I find examples that don’t quite fit. Occasionally, I’ll pop an example into the blog. Today’s example is a blog post by Gideon Golden with both an interactive stem&leaf plot of 2 letter Scrabble words, as well as a table of the same words, organized by first letter and last letter and color-coded by Cmglee:
How many ways are there to visualize a book? Bar chart, scatterplot, word cloud… that’s too narrow thinking. And, yes, there are websites showing how academics visualize text. But what happens out in the wild? Artists? School assignments? Professional designers? Statistics researchers?
Ever so curious, I decided to find out. To come up with some kind of method to search broadly, I picked one book, Lewis Carroll’s Alice’s Adventures in Wonderland and decided to find all the possible visualizations that might pop-up on Google/Bing text search, image search, scholar search. I found more than 40!
On the right are little teeny snapshots of the visualizations that I found. I won’t go into details on all of them, just a few highlights in this article, or you can view the video from the presentation I did for the Lewis Carroll Society of North America (lewiscarroll.org).
If you’re interested in more details, you can read the peer-review research paper. Some of the snapshots are cropped – the links to the full-size images are in the sources at the end of this post.
Visualizations 1-5 are from the visualization research community. Visualization #2 is a word cloud – only one word cloud of Alice in Wonderland is shown here even though hundreds exist. For the purposes of this article, I’m interested in differentvisualization techniques. Visualization #5 is Brad Paley’s TextArc from two decades ago – an early, wonderful, highly interactive visualization.
6-10 are visualizations from the digital humanities for analyzing text. I like #8, lining up adjectives for a character, providing a sense of the character. In this case Alice’s speech is described as soothing, piteous, or melancholy.
Visualizations 11-18 are from natural language processing. Interestingly, visualizations 15-18 have almost no words – even though they’re about a text.
Visualization #19 is a wonderful visualization from an art thesis by Yi-Chia Cheng. Paragraphs are converted phonetic sounds, shown as symbols using international phonetic alphabet, and stacked into distributions. Distributions can then be created and compared across languages to show how Alice sounds in different languages. (see Cheng’s thesis for many more distributions across languages).
What happens when looking a bit further a field than linguistic research and data analysis?
Visualization #20 is an artistic tool for drawing using sentences from text by Travis Kirton. In this case, an artist has drawn a figure of the caterpillar smoking his hookah using the corresponding sentences from Alice – creating a figurative, non-linear reading of that text.
Visualization #21 is digital micrography – that is – text which has been flowed to fit into arbitrary shapes. Lines of text are curved, bent and sized to follow the predominant flow of the shape. This particular example is from the PhD thesis of Ron Maharik, who automated the technique for even complex shapes such as puzzle pieces for floral shapes, such as this tiny portion from Alice (see figure 10.1, page 68 for the full image).
22-25 are timeline visualizations, some showing changes in Alice’s height over time. 23 includes Freudian analysis in relation to Alice’s height changes, mapping Alice’s psychological development over the course of the book.
Visualization #26 shows only a small portion of a small multiple visualization, showing 20 instances of Alice’s dress from across many publications and movies by Claire Wenzel. Who knew Alice had so many dresses, and an analysis of the fictional representation of Alice’s dresses over time can provide a view on our own changing society.
Visualization 27-28 are interactive physical visualizations, with flaps, tabs and pop-ups.
Visualizations 29-41 are even more broad examples from across the Internet. Some are borderline visualizations, but do use visualization techniques. #29is a list of color-coded places, characters and events. #30 is an infographic providing context to the book as well as content analysis.
#31 is a social network of characters from Alice in Wonderland. Each character is shown with an original illustration from Tenniel. The social network is shown by the lines joining the characters. Along each line is a sentence of text describing the relationship between the characters. Interestingly, this visualization is authored by a costume website — presumably knowing a bit more about the characters and their relationships helps rents more costumes.
#37 is a wonderfully hand-drawn homework assignment, with keywords in heavy marker underlined and rotated as well as lightweight sentences.
#40-41 are unique editions of Alice, with text layout changing, font sizes, caps, etc., modified by the designer in relation to the semantics of the text. Note the call out in #41 overlaying one of Carroll’s logical inversions to form an X.
That’s 41 visualizations. What can be learned from these? In the wild, there’s a lot more text on the visualizations than the research visualizations. And more use of typographic enhancements such as bold, underline, italics and so on.
* * *
These in-the-wild visualizations spurred me to create a number of other visualizations of Alice in Wonderland. Some of these are in my book Visualizing with Text in more detail (Routledge, Amazon, companion site). Large size versions of these images are available in this PDF, CC-license so available to use in teaching, etc. (Also embedded at the end of this post).
Visualization #42 and 43 are sub-word visualizations, indicating properties on syllables.
#44-50 are about words, typically extracted attributes about characters. For example, #49 lists adverbs associated with characters, with font-weight indicating most frequent descriptors – Alice is timid, the Queen is furious, the Hatter is dreadful.
#51- 56 are visualizations of phrases and sentences. #52 shows connections of repeated words from the Mad Tea Party. There’s a huge amount of repetition among the characters, reinforcing their position against Alice.
#55 shows the chapter title and portion of the first sentence for each chapter. Various metrics are shown — the underlying bar indicates the dominant emotion for that chapter as extracted using natural language processing. Chapter 6, Pig and Pepper is highly disgusting; whereas the Chapter 3, A Caucus-Race and a Long Tale is measured as sad.
#57 and 58 are visualizations of the entire book. They could be readable printed out on a poster. #57 has large red text under longer paragraphs. The large red text is a capitalized noun and an uncommon verb, adjective or noun in that paragraph – such as: “Rabbit rabbit-hole”, “Mouse lesson-book”, “Bill roof”, “Duchess frying-pan”, “Queen quarrelling”, and so on. The idea is to form large scale landmarks in the text to easily locate portions of the text. Even larger behind the text are the chapter numbers and titles in yellow.
#58 is a version of the entire text of Alice where the text is increased in size if it has been quoted on the Internet. After collecting and processing 200 quotations, the most famous quotes from Alice stand-out larger than the surrounding text. You can immediately see the most quotable quotes, and step closer to read the surrounding text. Interested in what’s the largest text?
“Who in the world am I? Ah, that’s the great puzzle!” (Alice, Chapter 2)
“We’re all mad here.” (Cheshire cat, Chapter 6)
Sometimes it’s important to think outside of the box of word clouds and bar charts: there is so much more possible and feasible.
Yes, there are more, so I see from responses on Twitter and elsewhere. 59-61 are some NLP visualizations: 59 creates little squares, one per sentence, brightness by sentence length. 60 transforms words to a vector space and plots, 61 isn’t quite a word cloud. 62-64 are more artistically driven: 62 animating sentences, 63 punctuation only, 64 is words inside large words which in turn forms a rabbit. I would not have though one could quite manage to get the layout of words to clearly form letters of larger words – apparently it’s quite feasible.
A Wonderland of Data Visualization
I did a presentation for the Lewis Carroll Society of North America (LCSNA) titled “A Wonderland of Data Visualization.” This presentation is more accessible to a wider audience and should be available on Youtube under LCSNA channel.
LCSNA is aware of additional visualizations: 65 is a set of interconnected bar charts comparing content from the original Under Ground vs. Alice’s Adventures in Wonderland – note that the first 5 chapters are largely the same, the latter chapters are largely new content. 66 is a similar analysis presented as a table, which is a type of visualization. 67 is another variation on a timeline indicating Alice’s height chart, in this example using Tenniel’s original illustrations and a related table. 68 is also a timeline in the wild, in this example, a highly illustrated timeline with short captions.
Senghor, L. Alice’s Adventures After Wonderland: Visualizing Alice in the Digital Era. Visual Learning: Transforming the Liberal Arts Conference, 2018. See also: slideplayer.com/slide/3575003 and kateogorman.org/text-analysis/voyant-tools
Jacques Bertin discusses texture in visualization, in his landmark book Semiology of Graphics (1967 original French edition, 1983 English translation). Much of modern visualization theory and implementation hinges on Bertin’s framework, including the notion of visual attributes, marks and layouts (for example, Wilkinson’s Grammar of Graphics, or Bostock’s D3).
Bertin talks about texture as a visual attribute and shows quite a few examples of texture applied to points, lines and areas. Bertin’s work exists at the heyday of Letraset – transfer film used by graphic designers to create professional graphics. Letraset provided professional fonts, graphics and textures that could be applied by any designer to create professional graphics for use in glossy magazine ads, to black-and-white screen graphics in newspapers, down to small scale zines – it was the democratization of desktop publishing before computation desktop publishing appeared in the mid 1980’s. Letraset and their competitors offered a wide variety of ready-made textures:
And these kinds of textures appear in Bertin’s lines and maps and substantially influenced his work. Note how textures can be combined together to represent multiple variables:
But what is texture? What are the variables that make up texture? What are the parameters for use? Did Bertin articulate all the possibilities, or are there more?
This is too big of a question for a single blog post, so instead, consider a very narrow use of texture as applied in only one dimension – that is – a texture applied to a line. Texture constrained to a line is very limited, particularly if the line is understood to have no width (at least for this blog post).
Dash Patterns: Length, Gaps and Rhythm
Like Bertin, we can think of texture as a sequence of on/off values applied to a line to create a sequence of dashes. The regularity of the sequence allows the viewer to distinguish between a line with a short dash pattern with big gaps and or long dash pattern with small gaps, such as A50 and F90 in the image below. The wide range in variation indicates that these lines could be used to show quite a few different categories within one visualization. Also, both the dash length and the the proportion of ink could be utilized to represent quantitative values associated with the line, e.g. less ink for lower contour lines, more ink for higher contour lines; short dashes for low certainty, long dashes for high certainty, etc.
In the bottom half of the image, other variants are considered. Rhythm can vary: on the left, the lines have a consistent dash-gap rhythm ABABAB, as seen in line G0. In the center column, a dot is added into the gap: the line now has the rhythm ABCBABCB, such as G1. It has the same ink as its counterpart in the 0 column, and is clearly differentiated. The right column has two dots (rhythm ABCDCBABCDCBA – which is clearly easier to describe as a visual line shown in G2 rather than as an alphabetic sequence). Lines with none, one, or two dots are clearly differentiated, allowing for many more possible combinations dash sequences to be used.
Randomizing Lengths and Gaps
In the bottom half of the image, within each column the regularity of the line pattern is perturbed. That is, with each successive line a bit more randomness is added. In G0 the dash lengths and gaps are consistent, in H0 the difference is almost unnoticeable, and by K0 there is some noticeable differences in dash lengths and gaps. G0 is the same line as E70. The K0 line retains the highest similarity to line E70, even though dash lengths and gaps have been modified more heavily. The ability to make minor adjustments to dash lengths and gaps while retaining the same line identity is a property exploited by old mapmakers – essentially it is preferable for the dashed line to be solid at a corner, not a gap:
As seen on this Ordnance Survey map from 1900, the dashes are solid at corners. The draftsman (drafts-person) has made minor adjustments to the intervals to retain the dash style while making the point of change in line direction visually explicit. Another interesting adjustment is that the gaps in the dashed lines occur when a dashed line crosses another line, as can be seen on the dotted line crossing lines in the far left and far right in the above thumbnails. This facilitates visual separation of the lines, not allowing them to be become an intersecting blob.
These kinds of adjustments are critical to making dashed lines effective in visualization. Consider a timeseries chart using a dashed line. If the gap occurs at a high point or low point, then it is impossible to determine the value from visually reading the chart, significantly reducing the effectiveness of the chart:
In the left chart, the drafts-person has a solid dash at every corner, whereas the Excel chart on the right has arbitrary dash lengths and gaps leading to corners not being explicitly visible. Since the corners on a line chart represent the actual data points, the arbitrary rendering results in data points not being visible! Yes, it is feasible in Excel to also plot the data points (as boxes, circles, stars, etc), but that only clutters the chart and breaks the rhythm of the line style — the drafts-people didn’t need to add extra marks for each point.
Given that a data visualization is supposed to show data, it is strange that the line style pattern overrides the visibility of the datapoints. Therefore, implementing effective dashed lines for visualizations isn’t as simple as defining a dasharray in SVG or D3. A lower-level dashed line module would be needed to make sure solid dashes occur at corners. This is non-trivial: what if there are multiple sharp angles in a short sequence (e.g. on the 18th day in the 1912 chart above), or multiple line crossings?
Showing Rhythm with Transparency, Saturation, Brightness or Hue
In Bertin’s example, texture is binary: it’s on or off. Much more is possible using computer graphics as opposed to transfer film. Transparency, saturation, brightness or hue can be modified with the regularity of the dash sequence:
In the top block, L50 – Q90, gradient transparency varies from solid to transparent to create the same pattern as previously seen in the dashes on lines A50 – F90. The lines in the 50 column use a linear gradient that goes smoothly from min to max and back to min, whereas in column 70 the gradient drops off more quickly than it rises, and in column 90, the gradients have a sharp drop off. Interestingly, the non-equal application of the gradient appears to give the lines the illusion of motion, like motion blur in photographs.
In the bottom block, Rs – Wh, the same patterns are used with attributes of color. In the first column, only saturation is changed (from orange to grey) which is difficult to perceive as there is no contrast between the two colors. In the second column, only brightness is changed, with the alternating pattern of light orange, dark orange clearly visible. And the third column changes hue, resulting in a repeating rainbows along the lines. Again, attempting to use SVG and D3 to implement these is difficult and the approach used in my quick code won’t work with curving lines.
More importantly, these examples suggest that texture is operating at a different level than visual attributes such as length, transparency, hue and so forth; as the texture can be represented in any these other variables. So Bertin’s notion of texture and how texture fits into the formal definitions of visual attributes of data visualization may be more nuanced than the current models used in visualization research.
Conjunction of Textures
Bertin uses multiple overlapping textures to convey multiple variables in his maps: textures of small dense dots, with large coarse dots, with lines oriented in one direction, and lines oriented in a different direction. Can multiple dash patterns be combined on one-dimensional lines? Yes:
In the top row, the line X1 has a very short pattern with a lot of white space is shown. Then, on the left, a line Y2 also has a short pattern and an interval that is a multiple of the interval for X1. The two patterns can be interleaved form the line XY12. The right image from Bertin back at the top of this article uses this interleaving appproach: it uses carefully designed textures such that dots fit neatly between the diagonal lines.
On the right, the first line X1 is the same short dash pattern, while the second line Y7 has a long dash pattern: there is no way to combine the two patterns without intersection. Here the patterns are combined to alternate on/off to create a reverse video effect (i.e. the patterns are XORed together).
This kind of approach might be desirable for charts with many different dimensions, for example, a census chart plotting unemployment overall/minorities, and for all ages/and those under age 25. Thus lines can be overall solid, minorities short dash; all ages solid (same data as overall), under 25 long dash, and the conjunction of minorities under 25 in the conjunction of the two dash patterns.
If dash patterns are problematic, then why use them at all? Sometimes there may be a need to use many lines, more than can be comfortably differentiated using color. Dashed lines are also common on line charts to show predicted data, or on maps to show unpaved paths: semantically a dashed line can effectively convey the data has uncertainty. Dashed lines have real uses.
Texture is an under-explored area of data visualization. Historical charts and maps do show that these can be effectively used. Visualization tools and libraries, however, use dashes arbitrarily and don’t take into account how to draw dashes to suit perceptual needs. Furthermore, our definition of the design space around texture may be somewhat lacking – perhaps some future grad student will want to take on some of these issues.
I’m really appreciate some of the examples I’m seeing in the wild. Here’s some fantastic examples from Georgios Karamanis. I like the shifted + bold text indicating voting topics at the UN, and word-pairs describing makeup. And Georgios provides the code, so if you’re into R and want to see how to implement some of the text visualization techniques from the book, see his github.
Also exciting to see an endorsement from Michael Friendly, and many others on Twitter. Thanks for the posts.
I’ve talked about the book internally at our company (Uncharted) and have been pleasantly surprised to see some of the ideas weaving their way into some of our visual analytics, such as a button indicating the color legend within the button glyph; or a technique for interactively labelling neighbourhoods while zooming around a massive network.
While I can’t really do a book tour during Covid, I did a talk at Naomi Robbins’ Data Visualization NYC Meetup and interview with Lee Feinberg’s Analytic Stories. Looking back at the videos, I see I may have talked over a couple people – sorry! Happy to follow up.
This technique of fitting type onto lines or into shapes has been going on for centuries: I like Calligrammes from the early 20th century, medieval monks speaking in scrolls (on the cover of the book!), text set into the shape of an axe in a book from 1530, or awesome psychedelic posters, such as Wes Wilson‘s posters from the 1960’s. For any visualization researchers interested in algorithms for fitting text into complex shapes, see Ron Maharik’s Digital Micrography research and PhD thesis.
News headlines about the GameStop price swings this week reminded me of some old SVG visualizations of stock bubbles and crashes that I’ve done. SVG was around before D3. I generated SVG visualizations in the mid-2000’s well before D3.js. It was painful in comparison, but at the time it was a lot of fun.
The objective was experimentation: what could be done with a scalable vector graphics library? As such, it wasn’t constrained to screen resolution and screen dimensions (which at that time was typically 1600 x 1200) – rather you could do much more detail, import into Illustrator and print things at very high resolution with lots of detail, lots of transparency, and tiny text.
Here’s an example: Microsoft’s daily stock price from late 1992 to early 2004.
That’s 12 years of daily data with about 250 trading days per year resulting in a 3000 px wide visualization — which now draws just fine on my 4k display. With SVG, it is easy to layer in many different analyses creating different marks, lines and areas. Here’s a closeup of Lucent’s daily stock price during the Internet bubble:
Many different graphical marks indicate data and derived indicators:
The blue line indicates the daily price.
The small green/yellow/orange/red bars behind the blue price line indicate the monthly price move from the beginning to the end of the month, colored by whether the price increases (green) or decreases (red). It aligns neatly with the monthly grid, filling in stripes within the grid.
The fat green/yellow/orange line behind that indicates yearly change in price. (There’s also light grey boxes behind that align with the thicker yearly grid.)
The green/purple circles indicating successive high/low points. Values for the highs and lows are indicated in text as well as the date. All successive low points are connected by a straight purple lines, all successive high points are connected by green straight lines. The zone between the green/purple lines form an envelope around the price range.
The many orange lines are moving averages each with a different time period, forming a guilloché. When moving averages start to cross it is indicator of a change in trend, e.g. from up-trend to down-trend (as many stock traders know). In 1997, the averages start to converge but then the trend continues. However, in 2000, the moving averages successively start to cross, and by June most of the moving averages have crossed, before the much steeper crash.
The large arcs and corresponding fills indicate major trends. Up-trends start at trend low to the ending trend high (e.g. 3/19/1997 at $10.75 to 12/8/1999 at $71.31: up almost 7x!) Down-trends start at trend high and end at trend low (e.g. June 5, 2000 at $56.23 to a point outside the closeup: 9/27/2002 at $0.77: down to almost 1% from the start, ouch!) This is what a bubble looks like and how it plays out.
The filled shadows in pink and blue indicate the 52-week low price and 52-week high price. On the first two-thirds of this chart, the pink shadow is predominant as the stock keeps going up with a few “lakes” of blue filled in the occasional dips. In the last third of the chart, it’s almost entirely under the blue shadow as the stock tanks.
Why is it important to have all these different markers, bars, lines, arcs, text, and so on?
There are many techniques to analyze timeseries data – moving averages, standard deviations, envelopes, and so on. In finance, one can explicitly study timeseries analysis. And more broadly, these analyses apply beyond stock prices to electrical grid loads, network utilization, software performance, automobile diagnostics, and so on.
SVG is scalable, so I applied it to longer and longer timeseries. Here’s the Dow Jones Industrial Average(R), when I re-ran the code with in 2010 to compare the 2008 financial crisis to the 1929 crash:
So, 114 years of daily data results in a plot more than 28000 pixels wide. That’s more than my 4K screen can display in it’s entirety at full resolution. But paper can. The visualization uses a log scale: you can see the magnitude of the 2008 crash at the top right is far smaller compared to the crash of 1929. I.e. from the index peak in 2007 around 14000 to a low of 6500 in 2009, the index lost a bit more than half of it’s value; whereas in 1929 the index peaked at 381 in September 1929 then dropped to 44 in 1932, down 88% of its original value! There was a lot of pain in 2007-8, but the massive intervention in the markets by the Federal Reserve and central banks helped stave off a much bigger crash and avoid a much bigger, longer recession.
There’s a few other things going on with this experimental visualization. Here’s a closeup of the 1950s:
In addition to the the many different shadings and markers, the grid lines also participate in indicating data range: rather than extend across the display, the grid lines are localized to the line +/- a range. Axis labels also follow data-driven rules. The price labels in this snapshot follow the grid (note the stock market high of 381 in 1929 wasn’t surpassed until 1954, some 25 years later). The date labels are completely data driven – indicating dates on the top side of the line when the price hits new highs, and on the bottom side of the line when the price hits new lows.
Why is it important to have such long timeseries in so much detail?
Our firm has clients with financial timeseries that are more than 200 years long. Seeing all the detail is important. The current GameStop bubble is not unique, there have been many, many more, going back to railroad mania in the 1840’s or the South Sea bubble in 1720’s. Different bubbles will play out in different ways: having detail allows for comparison to prior bubbles for insight into the current bubble. Recoveries from recessions will be different for different sectors. Some market experts will use this information to inform their portfolio strategies in response to GameStop, or in response to Covid, or in response to an election cycle.
And why the variations in grids, arcs and areas?
D3 is great and much can be done out-of-box. But, when you just use the standard examples applied to different data, you might not be indicating the things that matter to the end user (i.e. the purpose is insight, not pictures). As such, it’s important to use the toolset to experiment with the underlying graphics to highlight the core insights.
I’ve done much more experimentation with SVG before D3, but those will be for some future blog posts.