Simon Potter simon.potter@auckland.ac.nz and Paul Murrell paul@stat.auckland.ac.nz
Department of Statistics, University of Auckland
September 13, 2013
Abstract: The grImport2 package imports vector images that are stored in the SVG image format for inclusion within images generated by R. The original grImport package performs the same task, but instead using PostScript images as its source image format. This technical report describes the steps taken to improve not only the grImport package, but also follows the development of two new packages, grImport2 and grConvert. The latter, grConvert, converts images from popular vector image formats (PostScript, PDF and SVG) into valid input formats for grImport and grImport2. The grImport2 package is an entirely new package that does not suffer from some limitations of the grImport package, whose limitations are primarily due to the dependence on the PostScript image format.
The aim of this work for the Google Summer of Code was to improve upon the grImport package [1] for the R Project [2]. grImport is a package for importing PostScript images into R, which are intended to be used within plots created with R graphics. The original goal was to accurately recreate the state flags of the United States of America. The set of flags are already hosted online by Toby Dylan Hocking (at http://sugiyama-www.cs.titech.ac.jp/~toby/flags/index.html) and serves as a good set of test cases to see where grImport needs improvement.
One example of a state flag that needs improvement is that of Colorado. This is demonstrated in Figure 1 where the original flag is shown on the left, and grImport's attempt at importing and rendering the image is shown on the right.
What Figure 1 shows us is that grImport is not
parsing rectangles correctly. More specifically,
grImport does not understand
PostScript's [3] /rectfill
and /rectstroke operators and consequently it
cannot draw them. To correct this rather obvious flaw,
support for these operators were added to the
grImport package, which now renders many flags
correctly without any further changes necessary. The
corrected Colorado state flag is shown below in Figure 2.
Figure 2 shows us that indeed, this simple fix does correct the rendering of the state flags of Colorado. In fact, it greatly improves the appearance of most state flags. However, limitations in both the PostScript image format and R graphics mean that it is simply not possible for all of the state flags to be imported by grImport. An example of this is PostScript's lack of support for semi-transparency when painting within in graphics objects e.g. a path or a rectangle. Additionally, R graphics does not support clipping to paths, only to horizontally and vertically aligned rectangles. These are fundamental limitations that cannot be worked around by grImport and was the primary motivation for the development of the grImport2 [4] package.
To begin with, let us first demonstrate an example where the grImport2 package illustrates an improvement. This is shown in Figure 3 where the flag of Hawaii renders with a few errors in grImport. The key problems with grImport's rendering is that clipping is not applied either to the horizontal stripes or the boundaries of the Union Jack. The key reason why grImport does not perform clipping is because PostScript clips to a path, which is not possible with the R graphics engine. However, because grImport2 can work out the bounding box of a clipping path, it can also clip to the bounding box. The result is that the flag of Hawaii now renders correctly with grImport2, where it could not do so with the existing grImport package.
Despite the aforementioned improvements, there are some features of images that cannot be rendered in R graphics at all. We have already mentioned clipping to a path, but another example of this is gradient fills. A state flag that uses this feature extensively is the state flag of Kansas, where much of the central features of the flag are filled using linear gradients. This graphical feature is not currently supported by the R graphics engine, but it is supported by the gridSVG package [5]. The gridSVG package enables us to use features of SVG [6] within R, the key limitations being that the image must be a grid [2] plot, and that the resulting image must be an SVG image which is generated by gridSVG.
Figure 4 demonstrates that that if we support linear gradients, as is the case when we render with gridSVG, the state flag of Kansas can render correctly.
To summarise, while it is possible to further improve grImport's understanding of PostScript, further development of grImport is fundamentally limited by the features of both PostScript and the R graphics engine. These limitations are not present when using SVG as a source image format instead of PostScript. These are the primary motivations for the development of a new package, grImport2, rather than further extending the grImport package.
grImport2 targets the SVG image format, which
contains a superset of the functionality of the R graphics
engine. One of the key advantages of SVG is that it is a
grammar of XML [7], which means that it
is straightforward to parse with the XML
package [8]. For grImport2,
the use of the SVG format is similar to grImport's
use of its custom RGML image format, which is also a grammar
of XML. SVG and RGML are similar due to the fact that both
XML grammars attempt to describe how to draw an image. The
key difference is that grImport2 can import the SVG
image immediately, but grImport must first create
the RGML description (via PostScriptTrace()),
which can then be imported. Another key benefit of
targetting the SVG image format is that we not only have
complete access to the features of the R graphics engine,
but also the features of SVG when the gridSVG
package is used.
It may indeed be the case that PDF [9] supports all of the required graphical features, but it would still require the PDF images to be parsed in some way. It is this parsing step that is trivial with the SVG image format (with the XML package) but far more difficult with PDF. In order to parse PDF, we would need write a fairly large amount of code, especially considering most existing PDF libraries do not expose the graphical components of a PDF document. They instead choose to provide only a high level rendering function. Furthermore, even if PDF were even more graphically sophisticated, SVG is sufficiently capable of drawing visually identical plots to PDF.
A complication with SVG is that unlike PostScript and PDF it does not require the use of a physical coordinate system (e.g. points). This, among other things (such as attribute precedence), complicates the process of importing an SVG image. However, if we restrict the SVG we support to the SVG that is generated by the Cairo graphics library [10], then we know that the SVG images that we import are guaranteed to belong to a common coordinate system and other features of the image will be flattened, making it easier to parse. In particular, this means that grImport2 does not need to parse all of the SVG image format, only the subset of SVG that Cairo generates. Furthermore, because Cairo is is a supported graphics renderer for many open source libraries that import vector images, we know that it is possible for many input formats to be mapped to the Cairo SVG subset of SVG.
We have established that the task of importing vector images for grImport2 is greatly simplified when grImport2 imports only the subset of SVG that the Cairo graphics library draws. Additionally, it is not necessary to target one specific image format, instead we can leverage existing libraries that already handle the specific image formats [11] [12] [13]. However, grImport2's responsibility is for importing graphics, not for dealing with image formats. The grConvert package has been created to create compliant vector images for importing with either grImport or grImport2.
The grConvert package takes advantage of the fact that there are several mature, performant, and easy-to-use libraries for parsing popular vector image formats. Another key advantage of these libraries is that they support rendering by the Cairo graphics library. This means that we can take any PostScript, PDF, or SVG image and render it to any image format that Cairo supports. For the purposes of grConvert, it only ever needs to use Cairo to render to either PostScript or SVG for grImport and grImport2 respectively.
The interface provided by grConvert is minimal as
there is only one function of
note, convertPicture(). This function takes an
input vector image that is one of the supported formats
(PostScript, PDF or SVG) and exports either PostScript
for grImport, or SVG for grImport2. By
default the types of images that we import or indeed export
are inferred by their file extensions to make most
conversions as simple as possible.
To use convertPicture(), consider the cases
shown in Figure 5 where we convert various input images to
SVG, and also to PostScript.
R> library(grConvert)
R> # Convert PS to SVG
R> convertPicture("1.ps", "1.svg")
R> # PDF to SVG
R> convertPicture("2.pdf", "2.svg")
R> # SVG to SVG -- flattening
R> convertPicture("3.svg", "3-new.svg")
R> # Convert an SVG image to PS for grImport
R> convertPicture("4.svg", "4.ps")
This simple interface allows us to create supported input images for grImport and grImport2. For grImport, this was a task previously left as an exercise to the user, but is now much more convenient with the grConvert package. The convenience of converting between vector image formats in R reduces the time and effort needed to produce valid images for grImport or grImport2. Furthermore, in the case of grImport2, it also guarantees that the SVG produced contain the restricted SVG that grImport2 targets.
At this point in the technical report, we have demonstrated
some of the capabilities of grImport2, but have not
yet shown how this is performed. There are only two steps
necessary to draw images with
grImport2: reading, and drawing. Reading a picture
is performed using the readPicture()
function. It expects a filename or remote location that
points to an image that should be consumed to create
a Picture object. This Picture
contains all of the information necessary to reproduce the
imported image in R. Once the image has been read in
via readPicture(), we can reproduce the image
within R. An example of this is shown in Figure 6 where we
have a flag of New Zealand, which will be converted
by grConvert into Cairo SVG, and then read into R
by grImport2.
R> # Converting the image
R> library(grConvert)
R> convertPicture("nzflag-original.svg", "nzflag-cairo.svg")
R> # Reading it into R
R> library(grImport2)
R> nzflag <- readPicture("nzflag-cairo.svg")
There are two high-level functions we can use to draw this
Picture object. We will first
examine grid.picture().
grid.picture() is a function intended for use
with the grid graphics system. It takes
a Picture object, and enables it to be drawn
within any grid viewport. For example, we can draw
it to the entire page.
R> # 'expansion' is 0 so that we take up the entire page R> grid.picture(nzflag, expansion = 0)
One problem that we can observe from the rendered image in Figure 7 is that the lines in the Union Jack are not clipped at all. To correct this problem, grImport2 can clip to the bounding box of any clipping paths in the image. This is demonstrated in Figure 8.
R> # Ensure we clip to the bounding box R> grid.picture(nzflag, expansion = 0, ext = "clipbbox")
By clipping to the bounding box of any clipping paths, at least for this simple image, greatly improves the appearance of the flag. The rendering is not yet perfect, because the diagonal lines in the Union Jack are not clipped properly. However, we have reached the limitations of R graphics as it does not support this feature. We can leverage the gridSVG package to clip to these paths when it exports an SVG image. This is illustrated in Figure 9.
R> # Ensure we clip using clipping *paths*
R> grid.picture(nzflag, expansion = 0, ext = "gridSVG")
R> grid.export("nzflag-with-complex-clipping.svg")
The use of clipping paths instead of clipping rectangles improves the appearance of our flag so that it now renders identically to the source image.
We have focused on clipping, but there are further graphical features that we can use. For example, the central features of the state flag of Kansas contains linear gradients. These are not shown at all without the use of gridSVG, instead they are drawn using the current fill colour. This means that the use of gridSVG is beneficial, and in cases like these, essential for accurate reproduction of imported graphics.
R> convertPicture("kansas.svg", "kansas-cairo.svg")
R> kansas <- readPicture("kansas-cairo.svg")
R> # Use gridSVG features *and* gridSVG clipping
R> grid.picture(kansas, expansion = 0, ext = "gridSVG")
R> grid.export("imported-kansas.svg")
The second high level function that grImport2
provides for working with Picture objects is
called grid.symbols(). It is used for drawing
several copies of of a picture at a set of
(x, y) locations, with specified
sizes. For statistical graphics purposes, the most obvious
use for this function is to create custom plotting
characters.
To demonstrate how this may be used, we will use a custom
panel function for the lattice package. The panel
function will give us some (x, y)
locations to plot at and will also give a random size for
each of the pieces of the plot.
R> customPanel <- function(x, y, ...) {
R+ # Sizing the flags randomly for demonstration purposes
R+ sizes <- runif(length(x), min = 5, max = 10)
R+ grid.symbols(nzflag,
R+ x = x, y = y,
R+ default.units = "native",
R+ size = unit(sizes, "mm"),
R+ ext = "clipbbox")
R+ }
R> library(lattice)
R> xs <- rnorm(10, mean = 30, sd = 10)
R> ys <- rnorm(10, mean = 70, sd = 20)
R> xyplot(ys ~ xs, panel = customPanel)
Figure 11 shows that we have an easy way of using custom
data symbols in grid plots if we
use grid.symbols() instead of the
familiar grid.points() function. One drawback
however is that grid.symbols() can only draw
one
Picture object repeatedly. To draw multiple
custom symbols will require multiple calls
to grid.symbols().
One of the things you may want to do is change the appearance of a picture once it has been imported. There are two ways to do this, but we will first discuss the most convenient method; the second will be discussed later in this technical report.
The most convenient way to modify images imported
by grImport2 is to change their graphical
parameters, for example the colours that are being used
within a flag. Both the grid.picture() and
grid.symbols() functions have an optional
argument called
gpFUN. This argument should be a function that
takes a grid gpar object, and returns
a modified version of that object. This is a much simpler
approach than writing custom methods
for Picture and related objects (as is the case
with grImport). Primarily this is because the task
of creating graphics objects should be undertaken
by grImport2, but changing the
graphical parameters of pictures should be
configurable by the user.
As we have established earlier, the way in which the
graphical parameters can be modified is by the use of a
function that takes a gpar object and returns a
modified gpar object. We will construct a
function that desaturates all of the colours in
a gpar object, and apply that to our flag of
New Zealand.
R> # Create a gpar desaturation function
R> gpDesaturate <- function(x) {
R+ if ("col" %in% names(x))
R+ x$col <- desaturate(x$col)
R+ if ("fill" %in% names(x))
R+ x$fill <- desaturate(x$fill)
R+ x
R+ }
R> grid.picture(nzflag, expansion = 0, ext = "clipbbox",
R+ gpFUN = gpDesaturate)
Figure 12 shows how the gpFUN argument can be
used to modify the appearance of an imported image. The flag
of New Zealand now appears in greyscale colours. Not only is
it possible to change the colours of an image, but it is
also possible to manipulate the image by showing only the
strokes in the image.
Many of the examples used so far appear relatively simple,
perhaps with the exception of the state flag of Kansas
(Figures 4 and 10). However, because the software libraries
that grConvert relies upon are sufficiently mature,
the input images for grImport2 can be equally as
sophisticated as the images from which they were sourced.
For example, a type of picture that typically contains a lot
of content are pages in most PDF documents. We will
demonstrate that grImport2 can render complex
pictures using using a page from the grImport
article published in the Journal of Statistical
Software [14]. Figure 13 also
illustrates the use of the pagenum argument
in convertPicture(). pagenum
selects the page that will be converted from the source
document to PostScript or SVG.
R> # Convert page 2 to SVG for grImport2
R> convertPicture("grimport-jss.pdf", "complex.svg", pagenum = 2)
R> # Read in the page
R> page <- readPicture("complex.svg")
R> # Draw page
R> grid.picture(page, expansion = 0)
When rendering to the PDF graphics device, one can zoom in and observe that the image produced by grImport2 is imperceptibly different to the original page in the PDF document. The main reason why the content of the page is reproduced with a high degree of accuracy is because the Cairo graphics library translates text into an equivalent representation using SVG paths. A consequence of this translation is that grImport2 cannot import text as text, instead only the shapes are imported.
Much of the content of this technical report has discussed the motivation and the benefits to using grImport2, but this does not necessarily mean that grImport should be discarded. There are features that grImport has that grImport2 lacks. One of these features is the ability to import text as text. This means that it is possible to import a PostScript image containing text, then modify not only the text itself, but also its appearance (e.g. by changing its font). grImport2 can only import text as a path. This approach is problematic because paths are more difficult to modify and therefore generally support modification only to their graphical parameters.
Another advantage to grImport is that the R objects it creates to represent the imported image are far simpler, being comprised only of paths. This means that it is far simpler to modify an image if we know how to modify a path, rather than dealing with the multitude of groups and other types of objects that grImport2 creates. To demonstrate this, first consider the simple subset operation to show only a specific set of content in Figure 14.
R> PostScriptTrace("page27.ps")
R> page27 <- readPicture("page27.ps.xml")
R> survivalPlot <- page27[c(3:16, 18, 27)]
R> # Show survival plot
R> grid.picture(survivalPlot)
The subsetting operation shown in Figure 14 was simply reducing the image to only show paths 3—16, 18 and 27. Because everything drawn in grImport is a path, and the imported picture is simply a series of path drawing operations, the subsetting operation is simply keeping only some of the paths.
Performing similar manipulations of an imported picture
using grImport2 is more complicated
process. Firstly, as is the case with grImport we
do not have any convenient way of knowing, for example,
which paths in the flag of New Zealand represent the stars
or the Union Jack. A second complication is that there is
some structure to the Picture object that
closely matches that of the source Cairo SVG image. This
means that there are often trees of graphical content
represented in Picture
objects. grImport does not suffer from this
complication as it manages a series of drawing operations,
rather than any trees of graphical content.
Despite these complications, and after some trial and error,
we can subset or add content to
a grImport2 Picture object. Figure 15
shows an example where the Union Jack on the flag of New
Zealand is removed using subsetting operations on
a PictureGroup object.
R> # Create a copy R> nzstars <- nzflag R> # Remove pieces that represent the Union Jack R> # (children 3-8 of the main group) R> nzstars@content[[1]]@content <- nzflag@content[[1]]@content[-3:-8] R> grid.picture(nzstars, expansion = 0)
In general when modifying Picture objects, it
may be useful to know that they closely match the structure
of the SVG image that they were sourced from. For example,
when modifying the flag in Figure 15, we knew that the image
was composed of a single
PictureGroup object, which contains many
children. With this information at hand, it is reasonably
simple (albeit trial and error) to work out which children
of the PictureGroup are responsible for drawing
the Union Jack. However, it is clear that grImport
does make this type of task much simpler.
Another advantage of grImport is that it should be expected to work with any PostScript image, whereas grImport2 is dependent not only on SVG, but the specific subset of SVG that Cairo generates. grImport2 relies heavily on the structure of Cairo's SVG output so any change to Cairo could cause grImport2 to import the image incorrectly, or perhaps fail altogether. This is an important dependency to note but the Cairo SVG surface appears to be quite stable. It is unlikely to change in such a drastic manner that it would invalidate much of the parsing behaviour of grImport2.
In summary, if the goal is to accurately reproduce images in
R then grImport2 is a more complete
solution. However, grImport is simpler and
its Picture objects can be manipulated more
easily; if that is of greater importance
then grImport may be a preferable solution. The
cost of using grImport instead
of grImport2 is that reproduction of images is
unlikely to be as accurate.
We have demonstrated work performed as part of the Google Summer of Code for the R Project. This work resulted in minor changes to the grImport package, often leading to significant improvements in rendering. The grImport2 package was also created in order to work around limitations in the PostScript image format, in addition to limitations in R graphics. It is able to reproduce imported images with high fidelity, and when used with the gridSVG package, images can be reproduced to a greater degree than what R is capable of drawing.
In addition to developing improved tools for importing vector graphics, the grConvert package has been developed. It provides a convenient R interface for converting popular vector image formats into a format suitable for use in both grImport and grImport2.
The work demonstrated in this technical report was undertaken as part of the the Google Summer of Code 2013. Google funded the work entirely and without their support the development of this software would not have occurred.
The original versions of the flags shown in this technical report are freely available and have been sourced from the Wikimedia Commons. Thanks for the authors of these flags for providing such good examples for test cases.
This document is licensed under a Creative Commons Attribution 3.0 New Zealand License.
The code is freely available under the GPL. The described functionality of grConvert, grImport and grImport2 are present in the latest revisions of the packages on R-Forge.
In addition to the sources for the technical report and the packages described in the technical report, there is also a web page that shows and describes the improvements made to the rendering of the state flags of the USA. The web page is available at: https://dl.dropboxusercontent.com/u/54315147/import/state-table.html, with the source code for generating the web page available on GitHub at: https://github.com/sjp/grImport-flag-tests.