Canvas vs SVG: drawing a Wikidata graph

Intro

0:00 Hello and welcome back to episode ten, where we're going to load a real graph into smollerGraph, the mammals and birds on Wikidata, and give its page a canvas and a more serious look. Along the way, we measure what a query and a frame cost. Let's begin with a quick look at where we're going to be by the end of the episode.

0:20 By the end, smollerGraph is going to hold the mammals and birds on Wikidata, and its page is going to draw what a query returns on a canvas. This query asks for the carnivores, from each species up to the order, Carnivora, with each species, its genus and its family in a colour for each label. The line at the bottom says what the query cost on the server, and how long the page takes to paint a frame. And a search finds the red fox by its name, with its record as it's kept on disk.

0:50 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and each episode has a page here, with its chapters and its code.

Last time

1:04 Now, before we get to today's work, let's quickly go over what we did last time. Nodes and relationships got properties, each in a record of its own, in a chain. The facts we load got properties in braces, written as in Cypher, and Cypher got WHERE, with each filter planned as early as it can run. And the page's inspector shows a node's properties as they're kept on disk. But our graph is still ten nodes, four of them animals, which any database answers quickly, and any page draws in an instant. Today, we load a real graph, and see what holds up.

Today

1:40 Today has four parts, from the records on disk up to the page. Our records get room for the names a real graph has, and nodes get a label, which says what kind of node each one is. Then we load the mammals and birds on Wikidata, with their genera and families, about twenty-two thousand taxa. The page we built would try to draw all of them, so we redo it, to draw what a query returns, on a canvas, with a more serious look. And we measure how a canvas compares with SVG, which is what the page drew with until now. All right, let's get started with what Wikidata holds.

Wikidata

2:17 Wikidata is the knowledge base that Wikipedia and its sister projects share, with an item for each thing, and statements about it. Its data is under CC0, so it's free to use, and to keep in our repository. The red fox's item starts with its name in each language, and its statements come below. Its taxon name is its scientific name, the genus and then the species, in Latin. Its parent taxon is the genus Vulpes, whose parent taxon is the family of dogs and foxes, and so on up to the class, the mammals. That's a tree we can load as relationships. And its conservation status, from the IUCN, the International Union for Conservation of Nature, is least concern.

Room for real names

2:59 Before we can load it, a few things don't fit. Names run up to forty-two bytes, such as that of a tufted-tailed rat named after a reserve in Madagascar, and our records keep twenty-three, so about twenty-eight hundred of the names don't fit. Scientific names run up to forty-two bytes too, and a string property keeps fourteen, so about sixteen thousand of them don't fit. A species, a genus and a family are different kinds of node, which Cypher tells apart with labels, which our nodes don't have. And the page asks for the whole graph and draws each node in it, which is fine for ten nodes, and isn't going to be for twenty-two thousand.

3:37 The way to fit the names is to give the records more room. In graph dot h, the node record's comment adds its label, which says what kind of node it is. The name gets forty-four bytes, so it holds forty-three and the zero at its end, and the label gets twelve. With the first relationship and the first property, that's sixty-four bytes, twice what a node took. The property record's key gets sixteen bytes, and its value forty-three, so a string can be forty-two bytes long, and a property takes sixty-four bytes too. The static asserts check the new sizes, and their comment says why. A node and a property take sixty-four bytes, room for the names and values of a real graph, and a relationship still takes thirty-two, as nothing in it has grown. A page holds sixty-four nodes now, where it held a hundred and twenty-eight, so a graph's nodes take twice the pages they did. We could have kept the records small, with a long name in records of its own, as Neo4j does with long strings. A name in records of its own is a chain to follow, though, and bigger records keep a name one read away. K max label is the longest label that fits, worked out like the name's, and the comment over the limits adds the label. Set label gives a node its label, such as Species, in place of the one it had, as its comment says, and label reads it back, as name does. In graph dot c c, Add node starts a node with an empty label, as Copy field fills the field with zeros. Set label reads the node's record, copies the label in, and writes the record back, as Set property does with its first property. The name index keeps names as long as a node's, so its entries take forty-eight bytes, and a page of it holds eighty-five, where it held a hundred and twenty-seven. In graph test dot c c, keeps long names and a label adds the longest English name of a mammal on Wikidata, that of the tufted-tailed rat from Madagascar. Its comment says it's forty-two bytes, and its scientific name seventeen, and neither fitted last time. It gets the label Species, and its taxon, Eliurus carletoni. Find has to find it by its name, through the index, and its name, its label and its taxon have to come back whole. And the inspector's test gives the fox the label Animal, and sees a record of sixty-four bytes, four lines of hex: the name, padded with zeros, the label, Animal, then the two ids, its first relationship and its first property.

6:02 Let's build it, and run the tests, from episode one on. All forty-four of them pass, the old ones on bigger node and property records, and the new one with its long name.

Labels

6:13 A label goes after a node's name in a fact, as Cypher writes it, after a colon, as in Rabbit, colon, Animal. In facts dot h, the comment says so, with properties in braces after the label. In facts dot c c, a part's comment adds a node's label, after a colon, before the braces, and a part gets a label, a view into the line, like its text. Read part's comment adds that a label has to fit in a node's record. A part starts with no label, as well as no properties, and the part that's split at its braces starts with no label too. Then, when what's left has a colon, the label is what comes after it, and the name is what comes before. An empty label, or one longer than k max label, is a part we can't read. And Add with properties gives the node its label, when the part has one. In Cypher, a label goes after a node's variable. In parser dot h, the node pattern's comment shows one, colon Animal, and says that its variable, its label and its properties are all optional, and the node pattern gets a label. In parser dot c c, parse node's grammar gets an optional colon and a label, after the variable. When a colon comes next, a word has to follow, or it fails, saying it expected a label, and the word is the node's label. In plan dot h, there are two new kinds of step. Node by label scan gives a row for each node with the label it's given, as Neo4j's operator of that name does. Its comment says what it costs: with no index on labels, it reads each node's record to find them. Has label keeps the rows whose node has that label, which EXPLAIN is going to show as a filter. In plan dot c c, the comment over Add filters says it's given the step that found the node, the one that put it in a slot. It adds a filter for the node's label now, before the ones for its properties and for its comparisons. They still come as early as they can, so the rows they drop are expanded no further. And the comment ends with what's left out: a seek has found the name already, and a label scan has found the label. So Add filters takes that step, found, in place of whether it was a seek. It adds a filter on the node's label first, unless a label scan found the node, which checked the label already. After a seek, the name's filter is skipped, as it was before, since the index found the node by that name. In plan query, the comment over the start says how it's found now. A start with a name is found in the index. Without one, a start with a label is found by its label, and without either, each node in the graph is a start. That's the order to try them in, from the step that starts with the fewest rows to the one that starts with the most. So a start with a label and no name becomes a node by label scan. Add filters gets the step that found the start, which is the last step added, and expand for each node after it. In Run, a node by label scan goes through each node in the graph, as a scan does, and skips the ones whose label isn't the one it's after. Each of the others gets a row, with the node in its slot. Has label keeps a row when its node's label matches, reading the node's record for it, as a filter on a name does. And in Explain, a label scan is written with its slot, a colon and the label, as Neo4j writes it. Further down, has label is written as a filter, with the slot, a colon and the label, the way a pattern writes it. In graph test dot c c, the facts test reads a label too, Animal, on the fox. In parser test dot c c, the start of the WHERE test is an Animal, and the test checks its label. In plan test dot c c, scans a label or filters on it plans the animals that eat plants: a label scan for a, an expand, and a filter on b's label. With a name too, the plan seeks, and the comment says why: the index finds a name more quickly than a scan finds a label, so the label is a filter after the seek. The run test's animals and plants get their labels, and matches labels asks which animals eat plants: the rabbit eats the clover, and the field mouse the wheat. And the animals get their labels, as they had in episode one's Cypher, and the comment at the top says where a label goes. The red fox, the rabbit, the field mouse and the tawny owl get Animal, after their names. Then lines of their own give the others theirs: the clover and the wheat are plants, the mammals and the birds are classes, and the forest and the meadow are habitats.

10:50 Let's build it, with labels in the facts and in the planner, and run the tests, the new ones among them. All forty-six of them pass, and with the animals loaded, EXPLAIN shows the plan for the animals and where they live. It starts with a label scan for the animals, then expands along LIVES_IN, and filters on the habitat's label. Run for real, it gives each of the four animals, with the forest or the meadow, the tawny owl last, in the forest.

Mammals and birds

11:18 The facts come from a script of our own, tools slash wikidata dot py. Its docstring says what it writes: each species of mammal and bird the IUCN has assessed, with its genus, family, order and class, its scientific name, its conservation status, its weight when Wikidata has one, and its Wikidata id. It adds that Wikidata's data is under CC0, so the facts can go in the repo, how to run it, and that it takes a few minutes. It's Python, with nothing but its standard library, so it runs wherever Python does: we import json, sys and time, and urllib's parse and request, for HTTP. The endpoint is Wikidata's query service, which answers SPARQL, the query language for data kept as triples, a subject, a property and a value, much as Cypher is for property graphs. The user agent says who's asking, with a link, as Wikimedia asks scripts to do. The classes are the mammals and the birds, by their ids on Wikidata, and the ranks we keep are from class down to species, each with the label its nodes are going to get. Batch is how many taxa go in one question, and its comment says why it's three hundred: with more, some answers take longer than the query service allows. Max name and max string are what fits in our records now, as their comment says. Ask runs a SPARQL query, and its docstring says it waits and tries again when the service is busy, and returns the rows. It encodes the query and the format it wants, JSON, as a form, and makes a request with our user agent in its headers. It tries five times, in a loop, and each time it posts the request, waiting up to ninety seconds, and reads the rows out of the JSON that comes back. When the service is busy, or the network is, urlopen raises an error, so it waits ten seconds more each time. It gives up after the fifth, saying the query service didn't answer. Item takes the id at the end of an entity's address, such as Q7377, which is all we keep of it, as the rest of the address doesn't change from one taxon to the next. Walk walks the taxonomy down from a class, a level at a time, as our own Walk does, returning each taxon's rank and parent, and leaving out subspecies. One question for all the mammals runs past the service's limit of a minute, so walk asks for a level's children, three hundred at a time. It asks with P171, the parent taxon, and P105, the rank, which is optional, as some taxa have none. A new child goes on to the next level, unless it's a species. Describe asks for what we keep about each taxon, three hundred at a time, and its docstring lists it. That's the English name, British where Wikidata has one, the scientific name, the IUCN status, and the masses, in kilograms, leaving out a mass with another role than an adult's, such as a weight at birth. A weight at birth would make a fox weigh about sixty grams, as the one mass Wikidata has for the corsac fox is its weight at birth. Found starts each taxon with nothing, and each batch of three hundred is one question. The query asks for the British label and the English one, each with a filter on its language, then each of the others with its property: P225 for the name, P141 for the status and P2067 for the masses. Each is optional, so a taxon without a mass still comes back. There's a row for each combination, so the British label is taken first, then the English one, and the scientific name next. Statuses are lowercased, as Wikidata writes some in capitals, such as Data Deficient, and each mass is collected, as a number. Keep decides what goes in the graph, as its docstring says: the assessed species, and the ranked taxa above them. Going up from a species, each ranked ancestor becomes the parent of the one below, skipping the ranks in between, such as a subfamily or a tribe. Quoted writes a string in single quotes, or in double quotes when it holds a single one, so a value with an apostrophe can't end its string early, as our lexer doesn't read escapes. Main checks it's been given a file to write. Then, for each class, it walks it, picks out the species it reached, and describes them. It keeps the assessed species and the ranks above them, describes those ranks too, and prints how many taxa it walked and how many it kept. Then each taxon it kept goes on the list of nodes, with its rank, its parent and what describe found. The comment explains the order of the file: classes first, then orders and so on down to species, so a node's parent is in the file before it. Then it picks each taxon's name, with a set of the names taken so far. A species goes by its English name, as we would say it, with a capital at the start, and the ranks above it by their scientific names. Each taxon gets the first name that fits, with no brackets or colons and no more than forty-three bytes, and isn't taken. The comment says why: two taxa with one name would be one node, so the second takes its scientific name, or its id. The file is written as UTF-8, as some names have accents, such as the Réunion grey white-eye. Then in the file, a comment at the top, saying where the facts came from and under what licence, and a line for each taxon, with its rank as its label. Each property goes in braces when the taxon has one: its taxon, when it fits, its status, its weight and its id. The weight is the mean of its masses, in grams, as a whole number. Its Wikidata id goes in last, as a string, and then the line is written, with the properties joined by commas. Last come the relationships, PARENT_TAXON from each taxon to its parent, which is each taxon but the two classes. In compose dot yaml, the container loads the Wikidata graph and serves it, in place of the animals, and the comment at the top says so. The README says where the graph comes from, and how to write it again.

17:06 Let's build sg, and have a look at the first lines of the file the script wrote. It starts with the comment, then the two classes, the birds and the mammals, each with its label, its taxon and its id, then the orders. We load all of it into a folder of its own, which takes a fraction of a second: twenty-one thousand eight hundred and thirty-four nodes, and a relationship to the parent of each, but for the two classes. So let's ask where the red fox is, three hops up from its species, which is the order of carnivores. Now for the page, which asks sg serve for the whole graph when it opens, about two point four million bytes of JSON, a record for each of the taxa and each of their relationships.

17:46 The page as we left it, now on the Wikidata graph, makes an SVG element for each of the nearly twenty-two thousand nodes, with its name, and for each relationship. D3's force layout moves each of them, on each tick. What it draws is a tangle of lines, with circles flying apart, at a few frames a second, while the browser lays out and paints each of those elements again.

Drawing what a query returns

18:09 So here's what the page is going to do instead. It stops asking for the whole graph, which no one can read at this size anyway. It draws what a query returns, the way Neo4j Browser does, and no more than five hundred of its nodes, where Neo4j Browser draws up to three hundred by default. The rows come back in a table, a thousand at most. It draws on a canvas, one element that the page paints pixels on, in place of an SVG element for each circle and each line. And the server says what each query cost, its time and the pages it read, which the page shows on a status line.

18:44 In api dot h, Graph JSON goes, as the page doesn't ask for the whole graph any more, two point four million bytes of it for the Wikidata graph. Walk JSON goes with it, as a query does what the walk form did. Names JSON stays, as the page's search is going to suggest names from it. Node JSON's comment adds the node's label, and the node each relationship goes to, which the page is going to use to draw a node's neighbours, when a node is expanded. It still says that an id the graph doesn't have is an error, as it did. Query JSON's comment says what it answers with now: the columns, and up to k max rows of the rows. Then the nodes the rows return, up to k max drawn of them, with the relationships between them, for the page to draw. It says how long the query took on the server and how many pages it read, and after EXPLAIN, it answers with the plan. K max rows is a thousand, and k max drawn is five hundred. In api dot c c, we include chrono, and unordered set, a hash set, for the nodes already seen, and the includes that walks needed go. Graph JSON, Walk JSON and the walk's limit go. Node JSON adds the node's label, and each relationship gets the label of the node it goes to. Query JSON starts a steady clock, and notes the pages read so far. Once the plan has run, the JSON starts with the milliseconds, the pages read since, and the number of rows. The rows go in as before, up to k max rows of them. Then the nodes to draw, which its comment describes: the nodes the rows return, each once, in order, with the relationships between them. Up to k max drawn of them go on the list, each once, as a set remembers the nodes seen. Each node goes in with its id, its name and its label, read from its record, for the page to colour it by. For the relationships, it walks each drawn node's chain, from its first relationship to the next, reading each one's record. It keeps a relationship whose other end is in the set of nodes seen, so it's drawn too, with its id, its two ends and its type, and the JSON ends. In server dot c c, the page's files are served with a header, Cache Control no cache. Its comment says what that does: the browser asks whether a file has changed before it uses its copy, so a reload picks up a new page. The routes for the whole graph and the walk go. In main dot c c, sg query gets a clock too, the way Query JSON has one. It starts the clock and notes the pages read before it parses, and takes the time once the plan has run, before the rows are printed, so printing them isn't counted. Then, after the rows, it prints how long the query took, and the pages it read, as the comment says. In api test dot c c, the tests for the whole graph and for walks go. Node JSON's test sees the fox's label, Animal. And Query JSON's test checks the nodes and the relationship the page is going to draw, the fox and the rabbit, and the EATS between them, but not the forest, which no row returns. Its helper, Without time, leaves out the milliseconds, which change from run to run.

A page on a canvas

21:53 The page is new, so its old files go, graph dot js and style dot css, and new files take their place. Index dot html is laid out like a console now. The header has the brand and a search, with an input and a list for suggestions. On the left, an aside holds the query box, a text area with the dog family as its first query, which the page runs as it opens, a Run button, and the history of queries run, as an ordered list. In the middle, main has three tabs, Graph, Table and Plan, as buttons in a nav, with the graph's active as the page opens. There's a section for each, with the canvas and a legend in the graph's, a table in the table's, and the plan as preformatted text in the plan's. The status line goes under them, for what each query cost. On the right, the inspector has the node's name, its label, a button to expand it, where it's kept, its bytes, and its two chains. The style sheet, page dot css, lays them out on a grid, in the series' colours, and hides the tabs that aren't active. View dot js is the graph view, for a canvas. Its comment says what it draws, and how a canvas differs from SVG: we paint its pixels a frame at a time, where SVG keeps an element for each circle and line, for the browser to lay out and draw. A node's radius is nine pixels, and names are written under the nodes while there are a hundred and fifty or fewer, or once the view is zoomed in far enough to read them. Colour is an ordinal scale from D3, which gives each label a colour. Its comment says the species get the series' chartreuse, each rank above them a colour of its own, and other labels the rest, in the order they turn up. Colour of gives a node without a label grey. Layout makes the force simulation, in a function of its own, which the page and the benchmark we write later share, as its comment says, so the benchmark times the forces the page uses. Its forces are like the page's, with shorter links, a weaker push and a smaller collide for small nodes, and x and y forces pull each node to the middle, as its comment lists them. The Graph View class draws on a canvas, and calls on select when a node is clicked, and on expand when it's double-clicked. The constructor keeps the canvas and its 2D context, which is what we draw with, and starts with no nodes or links, nothing selected, the zoom's transform at rest, and no frame time yet. The simulation comes from layout, with a handler for each tick and one for the end, and its comment says each tick paints a frame, and once the layout settles, the view zooms to fit it. Resize sets the canvas's width and height in the screen's own pixels, its size in CSS pixels times the device's pixel ratio, and draws a frame. Its comment says why: the canvas has as many pixels as the screen does, which is twice its size in CSS pixels on many laptops, so circles and text stay sharp. A resize observer calls it when the canvas changes size. The zoom's handler keeps D3's transform, a shift and a scale, for drawing, and draws a frame when it changes. The comment over it says which does what, as there's a drag after it too. Dragging the background moves the view, and the wheel zooms it, while dragging a node moves the node, which the simulation lets go of when it's dropped. D3's drag takes the node under the pointer as its subject. When a drag starts, the simulation warms up again, and the node is pinned where it is, with fx and fy. As it moves, the pointer goes back through the zoom's transform, and the node is pinned under it, from the middle of the canvas, where zero, zero is drawn. When it's dropped, the simulation cools down, and lets it go. The canvas gets the drag first, then the zoom, so a press on a node drags it, and anywhere else pans the view. D3's zoom would zoom in on a double-click, so that's turned off, as a double-click expands a node. A click calls on select with the node under the pointer, if there is one, and a double-click calls on expand. Node at finds the node under the pointer, and its comment says how: D3's find goes through the nodes for the nearest one within a radius. That's quick for a few hundred nodes, as it runs on a press or a click, not on each move of the pointer. The pointer goes back through the zoom's transform, as the drag's does, and find looks within twice a node's radius. Show shows the nodes and relationships it's given, in place of the ones shown, or alongside them with keep, as its comment says, and nodes that stay keep their places. A map of the nodes shown, by id, lets a node that's still there keep its position, and new nodes are copied, so the simulation writes its positions on the view's own objects. Links whose two ends are shown are kept, with a source and a target for D3, and the simulation gets the nodes and the links, and restarts, with alpha back at one. Fit zooms the view so the graph fills it, with a margin, and no further in than twice its size, as its comment says. It takes the extent of the nodes' positions, across and down, with D3's extent, which gives the smallest and the largest of each. The scale fits them with a fifth to spare, and the smaller of the two, so the graph fits both ways. Then it works out where the graph goes, x and y, from the extent and the scale, so the graph sits in the view's centre once it's scaled. The comment over them says why: the middle of the graph goes to the middle of the canvas, where the simulation's zero, zero is drawn. A transition eases the zoom over six hundred milliseconds. Select remembers the selected node, for the ring around it. Draw paints a frame, and its comment explains the order: the relationships first, as a path of lines and one of arrowheads, then a path for each label's nodes. Each path is one call to fill or stroke, however many shapes it holds. Stroking a path draws its outline, where filling paints its inside. It notes the time it starts, then sets the transform for the pixel ratio, clears the canvas, and applies the zoom's shift and scale. Then it moves the origin to the middle of the canvas, where the layout's zero, zero goes. The canvas's transform takes care of the zoom, so the nodes are drawn at their positions from the layout, as they are. One path gets a line for each relationship, from its source to its target, and one stroke draws them all, in a see-through coral. Another path gets a triangle for each relationship, an arrowhead at the edge of the node it goes to, as the comment says. Its angle comes from atan2, from how far across and down the relationship goes. Its point is a radius back from that node's middle, along the line, and the path moves there first. Its two sides go back seven pixels, either side of the line, and filling a path closes each triangle on its own. One fill paints them all, in solid coral. D3's group splits the nodes by label, into a map from each label to its nodes, and each label's circles go in one path. A move to each circle's edge keeps the arcs from joining up, and the path is filled in the label's colour. The selected node gets a ring around it, a little wider than a node, in the text's colour, with a path of its own, as it's stroked and not filled. It's drawn after the nodes, so it sits on top of them, and its line is two pixels wide. While there are a hundred and fifty nodes or fewer, or the view's scale is two or more, each name is written under its node, with an outline in the field's colour, so it reads over the lines. Last, the frame's time, from the start of draw to now, is mixed into frame milliseconds, nine tenths the old and a tenth the new, which makes a moving average, for the status line, as its comment says.

29:30 App dot js is the page around the view, and its comment lists its parts. It's a query in our subset of Cypher, run by sg serve, with its answer as a graph on a canvas, as a table and as a plan. Then a history of the queries run, a search for a node by its name, and an inspector. It imports Graph View and colour of from view dot js, and makes the view on the canvas, with inspect for a click and expand for a double-click, both of which come further down. Count writes a number of things, such as one row, or three hundred and seven rows, as its comment says, so the status line says one row, and not one rows. Literal writes a value as Cypher does, a number as it is, and a string in quotes. Its comment adds that the quotes are double ones when the string holds a single one, as Pallas's cat does, which would otherwise end its string early. Show tab marks the active tab's button, and shows its section, hiding the others. D3's classed sets a class when its function returns true, and takes it away when it doesn't, and clicking a tab's button shows that tab. The comment over the query says that running one asks for its answer and its plan together, so the Plan tab always has the plan of the query that's shown. The query form and its text area come first, then the history, a list of queries. The comment over frame timer says the frame time comes a second after a query's answer, once a few frames have been painted. Run makes a small function, ask, which sends a query to sg serve in the query string, with fetch, and reads the JSON that comes back. A query that starts with EXPLAIN asks for its plan and nothing else, and any other asks for its answer and its plan at once, with Promise dot all, which waits for both. An error goes on the status line, which turns coral. The plan goes in its tab, and after an EXPLAIN, the Plan tab opens, and that's all. Otherwise, the query joins the history, the rows go in the table, the view shows the nodes and relationships, and the legend lists their labels. The comment over the status line says what it shows: on the server, the time from parsing to the last row, and the pages read from disk, and in the browser, the time a frame takes to paint. When there are more rows than came back, it says how many it shows. The summary counts the rows, then the nodes and the relationships drawn, and last the milliseconds, to two places, and the pages. D3's timeout adds the frame time a second later, after stopping the one before. Submitting the form runs the query in the box, without the browser loading a new page. Control or Command and Enter runs it too, as in Neo4j Browser, as the comment over its key handler says. Remember keeps the queries run, newest first, each once, and its comment adds that clicking one runs it again. A data join turns the history into a list, and a click on an item puts its query in the box and runs it. Show legend writes a key for the labels drawn, each in its colour, as its comment says, from the set of labels the view's nodes have, sorted, with no label for a node without one. Show table empties the table, then writes a heading for each column, and a row for each of the answer's rows, with a cell for each value, and null for a missing one. As the comment over the search says, as a name is typed, sg serve suggests the names that start with it, from its index, and picking one shows that node and its record. The input's handler reads what's typed, and asks sg serve for the names that start with it, in the query string, waiting for the JSON. When the box is empty, there are none. Clicking a suggestion empties the list and the box. Then it runs a query for that node, by its name, and opens the inspector on it. The inspector's button expands the selected node, and the comment over it says that a double-click on a node does too, drawing the nodes its relationships go to. Expand asks sg serve for the node's record, and draws a node for each relationship in its chain, keeping the nodes drawn, and writes the legend again. Inspect shows a node's record as sg keeps it on disk, as its comment says. When it's given no node, there's nothing to inspect, so it returns. It selects the node in the view, asks for its record, and opens the inspector. The name goes in the heading, and the label under it, in its colour, or no label, in grey, when it has none. Then where it's kept, its bytes, and its chain of properties, each with its key and its value, written as Cypher writes it, and where its record is kept. Then its chain of relationships, each with its type and the node it goes to, and where it's kept too, as before. Last, the page runs the query in the box as it opens. The Caddyfile tells the browser to check each file for changes too, as Caddy serves them on a server. And the README says what the page does now. It draws what a query returns on a canvas, up to five hundred nodes, with the rows in a table, the plan, and what the query cost, and a search and an inspector.

34:27 Back in the browser, here's the page at the end of this step, which opens on the dog family, with a colour for each label in the legend: the species, their genera and the family. The Table tab has the rows, one for each species. The Plan tab has the plan: a label scan of the species, then an expand to the genus, with a filter on its label, and an expand to the family, with a filter on its name. Back on the graph, let's ask for the carnivores, four levels of them: each species, its genus, its family, and the order, Carnivora. The query returns four names a row, so the page draws the four nodes of each row. We run it with Control and Enter, and the status line says how many rows came back, the nodes drawn, the time on the server and the pages read, and how long a frame takes to paint. Then the search, which suggests names as they're typed, from the index, and picking the red fox draws it, and opens its record. Its sixty-four bytes are its name, its label, Species, and its two ids. Expand draws where it leads, which is its genus, Vulpes. And the history has each query we ran, to run again with a click.

What a query costs

35:36 Now that sg query prints what each query costs, let's ask the Wikidata graph a few questions, each in a fresh process, so each page it needs is read from the files. The red fox's order, three hops up, starts from a seek in the index, and reads ten pages, in a fraction of a millisecond. The species of the dog family start from a label scan, which reads each node's page, and their relationships, to find thirty-nine of them, in a few milliseconds. Neo4j could start from the family and follow its relationships backwards, which our nodes can't do yet, as each keeps the relationships that start at it. And the species that are critically endangered, a label scan with a filter on a property, which reads each species's chain of properties, in a few milliseconds too.

SVG or a canvas

36:23 To measure what drawing costs, we're going to draw one made-up graph in three ways, and count the frames the browser paints. Bench dot html is a page with nothing on it but D3 and a module, bench dot js. Its comment says what it does: it draws a made-up graph of people, each knowing two others, with SVG, on a canvas, or not at all, while the page's layout moves it, and counts the frames painted in three seconds. It imports the page's Graph View and layout, and the parameters say how to draw, and how many nodes. A random number generator from D3, with a seed, makes an identical graph on each run, as its comment says: people, each with two relationships to people picked at random. The relationships go on one list, two for each person, each to someone the generator picks. With a canvas, a Graph View on a new canvas element shows them, with nothing to do when a node is clicked. Otherwise, as the comment says, it's SVG as the page drew it until now, an element for each line and circle, which the browser lays out and paints, or no drawing at all, for the layout's own cost. The SVG version puts a group in the page's middle, as the layout's zero, zero is drawn there, and gives its links a source and a target, for D3's link force. It joins a line for each link and a circle for each node, unless it draws nothing. The layout gets the nodes and the links, and on each tick, it moves each line's ends, x one and y one, x two and y two, and each circle's middle, as the page did. That's the work the browser redoes on each frame. Last, the frames, counted from once the graph is set up. The browser calls request animation frame's callback before it paints each frame, so counting the calls counts the frames, as the comment says. Count frame adds one and asks to be called again, until three seconds have gone. Then it sets window dot result to the frames a second, the frames over the seconds, which bench dot py reads. Bench slash ui dot py drives it, and its docstring says what it measures. That's how many frames a second the browser paints while the page's layout moves a made-up graph, from five hundred nodes to twenty thousand, drawn in each of the three ways. Chromium opens the page through Playwright, Microsoft's library for driving browsers, and the frames a second at each size go in bench slash ui dot json. Then it says how to run it, with sg serve running, and uv fetching Playwright for the script. It imports json, Path from pathlib, and Playwright's synchronous API, which waits for each step. The page is bench dot html, on sg serve's port, and sizes and draws list what it tries. Main launches Chromium with a window, through Playwright. The comment says why: with a window, Chromium paints with the GPU, as a browser on a laptop does, and without one, it paints a canvas in software. For each size and each way of drawing, it opens the page with the size and the way of drawing in its query string, waits up to two minutes for the result, and keeps the frames a second. It prints each size's row as it goes, and then writes them all to the file. Git ignores the results, as it does those of the other benchmark, and the README says how to run it.

39:37 Let's run it, with sg serve serving the page, and nothing else busy on the machine. A Chromium window opens, and loads the page for each way of drawing, at each size, and each line is a size, with the frames a second for no drawing, for the canvas, and for SVG. Up to two thousand nodes, all three keep up with the screen, at about sixty frames a second. At five thousand, SVG drops to about twenty-four, and the canvas keeps about fifty-nine. And at twenty thousand, even the layout on its own manages about fourteen.

40:10 Here's the run as a chart, with nodes along the bottom. SVG falls away first, past a couple of thousand nodes, as the browser lays out and paints an element for each circle and line, on each frame. The canvas stays close to the layout's own pace all the way, at about twenty-eight frames a second at ten thousand nodes, against about eleven for SVG. Past five thousand, the layout is what's slow: D3's force simulation runs on the page's one thread, and works out the forces between twenty thousand nodes on each tick. So the canvas was the change worth making, and D3's layout is quick enough for the five hundred nodes a query draws, while a whole graph of twenty thousand wouldn't keep up, however it's drawn.

What's missing

40:52 A real graph shows what we haven't got yet. A node keeps the relationships that start at it, so finding the species of a family means scanning each species, where Neo4j would follow the family's relationships backwards. A label or a property in a WHERE scans each node too, where an index would go straight to them. On the page, the layout runs on the page's thread, which a worker could take off it, for graphs bigger than the five hundred nodes we draw. And the graph is still written by sg load, all at once, while a graph that's written to needs to survive a crash part way through a write, which is what a write-ahead log is for.

Wrap-up

41:29 So, today smollerGraph got a real graph, and a page that copes with it. Nodes and properties take sixty-four bytes, room for real names and values, and a node has a label, which facts and Cypher read, with a label scan to start from. The mammals and birds on Wikidata came in through a script of ours, about twenty-two thousand taxa, under CC0. The page draws what a query returns on a canvas, with a table, a plan, a history, a search and the inspector, and says what each query cost. And measured, a canvas keeps up with the layout long after SVG has fallen behind.

42:06 This is going to be an evolving project, so I'd love to hear from you. Let me know in the comments which animals you'd look up first, and if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.