Quarto integrated website system

1 Vibes

After giving it a thorough testing, I have feelings and thoughts about quarto.

Quarto’s strong ecosystem is a strong recommendation. There are vibrant discussion boards, many active developers, many active users, and good integrations into various IDEs (VS Code, RStudio etc). It has a corporate sponsor, Posit FKA RStudio, who have some extra add-ons they sell, some of which I am a fan of (Shiny, Posit Connect). I am on-record saying that a vibrant community is a better test of usefulness and predictor of future support for some software than my feeble, biased personal, aesthetic judgement. But read if for some reason you nonetheless desire to know my aesthetic judgement.

I can report that the resulting system is more tightly integrated than hugo is with blogdown, but not substantially simpler. I count that as a marginal win.

Quarto is more opinionated than blogdown if I use the built-in website system (although in principle I could build my own different website system). If I happen to like a 2-3 column layout blog with standard features (search, overview by date) everything is easy. OTOH, deviating from this layout is difficult and poorly documented. For example, it is not trivial to vary the CSS framework from the default Bootstrap.

The resulting websites are not high-performance for large website, as you might be noticing on this on. The are hefty, and slow to build. Since I am not a web developer but rather an academic, this price seems acceptable to me for my specific use case — the opinionated default is pretty close to what I want — but this might not be the optimal trade-off if your own needs differ.

There are some worrying signs of code chaos. Quarto websites will not win the The Grug Brained Developer seal of approval. The code is sprouting features at an alarming rate, for one thing. On the forums we learn that the code has some band-aid bits, e.g. there are two colliding template systems in use whose relationship is under-documented. A core developer has left the project and would like a minimalist holiday. Nonetheless, every system has its pain points, and I am willing to give this system a go. The code chaos is not yet worse than other systems I have tried.

The theming and site structuring is somewhat less flexible than hugo, the backend used by blogdown, but the integration with said backend is better, so is the overall experience is somewhat better on net, since much of the flexibility of hugo was useless to me in any case, hidden behind feature mismatch. By the same token, quarto leverages many more features of pandoc than was possible with blogdown, which leads to many well-supported advanced typographical features.

If one wished to use the quarto engine to experiment with weird alternate features (such as the content ranking, recommendation or the quirky indexing systems as seen on this site) then one is, AFAICT, out of luck we can use custom listings. That seems to do about 80% of what I want, albeit slowly. YOLO! Let us 80/20 it.

2 Community support

• quarto-dev/quarto-cli · Discussions about websites has an active community.
• mcanouil/awesome-quarto: websites is a good place to go to find worked examples of the less-documented stuff.

3 Quarto websites are slow to load

Quarto websites can enormous compared to the equivalent blogdown site. Browser memory usage by all the javascript wizardry etc is substantial. Even though they look like small and efficient static sites, the actual cost of all the dynamic behaviour adds up; Or we could say: quarto is probbly mostly thinking about tiny websites at this stage of its development, and big projects like the one you are reading now push its limits.

• Improve first load weight? Discussion #9330

The listings in particular, can be huge. When I migrated this website to quarto the front-page download went from less than 1MB to 135MB, which is, for reference, comically huge for a 3 paragraphs and list of the titles of the last 10 blog posts.

• Quarto does not thumbnail images — Discussion #9390
• Automatically reduce thumbnail image size on Quarto website — Discussion #3793
• Can listings dynamically load images? Discussion #9434 Fixed in latest quarto!

I assume that this is because the listing on that front page, in order to provide dynamic sorting etc, loads essentially all the posts on the blog, no matter how old, and their associated images, at full resolution. AFAICT quarto does not generate image thumbnails, and also AFAICT the listing system is was not built around lazy loading.

Potential partial solution via image resizing:

• danmackinlay/quarto-thumbnail: Try to reduce quarto listing bloat with thumbnail images
• A quarto extension that optimizes your images pre-render
• abhiaagarwal/optimize-images: pre-processes quarto figures and generates optimized versions

Those extensions did not actually work for me, so I wrote a custom script that postprocesses the site.

4 quarto websites are slow to build

tl;dr a typical CLI invocation of blogdown was less than one second, maybe longer if I was working on a complicated computation, say 1 minute. A typical CLI invocation of quarto render for this site takes about 12 minutes.

I am surprised how much I miss the speed and efficiency of blogdown, with its smugly high-speed hugo backend. I honestly thought the speed was not an issue until switching from blogdown to quarto website made my site muuuuuuuuch slower. To build the 1000+ posts on this site typically took hugo a few seconds.

There seems to be a lot of re-rendering bullshit going on. There are various tricks to make rendering go faster, such as caching the code execution, but ultimately quarto render is still slow, with even the most aggressive cache settings, compared to hugo.

AFAICT the problem is partly that the quarto website engine is slower than hugo, and partly that quarto is re-rendering everything every time (in the sense of converting markdown to HTML, not of executing code inside the markdown, which we can avoid by using the cache and freezefacilities). Blogdown+hugo was smart enough to only re-render the things it needed to, and reused the HTML from before, so there were few things it needed to. I think? Or maybe hugo is just much faster because it is a compiled binary that doesn’t arse around with javascript and pandoc and stuff. Or both. Knowing which will not change much for me so I will not investigate further for now.

UPDATE: according to a core dev

In generalities, our runtime is roughly spent 2/3s inside Pandoc, and 1/3 in Deno. Our Pandoc filtering infrastructure is pretty extensive, and some of our early decisions there have performance consequences that we didn’t foresee: we’re now working on them. In Deno, our performance profile is relatively flat, and so the work is going to be more of the “continued small fixes” kind.

I suspect the friction might be that the default quarto workflow favours a small number of immaculate, unique snowflake documents, where as I am more like a sit on the snow machine and make a blizzard kind of guy.

The simplest workaround seems to be to use quarto preview --no-serve, which only renders the recently changed things, and so is much faster than rendering 1000+ things. You might think that is a sufficient solution. Unfortunately not. Because:

1. quarto preview is still not that fast. It takes 32 seconds on this machine, on a typical invocation, to decide what part of this blog incrementally render, which is already 5 or 6 times slower than blogdown took to finish an incremental render and build.
2. I can leave quarto preview running, which amortises the start-up time, but this uses colossal amounts of RAM if the site is large; I guess it is serving the site from memory?
3. This quarto preview server process burns a surprising about of compute. It tanks my battery if I try to edit my blog on a long trip, much more rapidly than, for example, running a full-feature realtime audio workstation.
4. The file watcher gets stuck sometimes, and does not notice changes. This annoyance is minor, but real, because I waste attention each time it happens trying to work out if the problem is quarto or me. As an intermittent bug involving concurrency, I expect this one to persist for a long time, based on experience of similar bugs in similar projects.
5. Even if I leave quarto preview running it is still very slow, because it generates the index pages for this blog (which are large lists of 1000+ other blog posts) on every minor page change.
6. The precise degree of slowness is particularly corrosive to my personal attention span. 30 seconds is long enough to drag productivity to a halt, but not long enough to make me go and do something else.
7. A full re-render while running the preview server seems to lead to inconsistent and messy results, sometimes crashing the preview server, or leaving detritus lying around. So the server is not set-and-forget, but rather a thing I need to babysit, and make sure it does not clash with a full re-render.
8. The quarto server itself is not that amazingly fast at serving files, for some reason. It seems to block a lot and take ages to serve me a page, even if it has already rendered the page.
9. Sometimes, and I do not know why, the server decides a full-re-render of the site is necessitated, although I have done nothing, and suddenly previewing that one change is now a 12 minute wait.

quarto publish netlify --no-render --no-browser --no-prompt

5 Theming

Custom HTML theming is not too bad for simple CSS tweaks. Although the documentation is brusque, this part mostly “just works” in the sense that if I guess what to do it usually ends up being correct.

See

• HTML Theming
• More About Quarto Themes
• What layout do we support in the native themes? See HTML Page Layout
• I found myself additionally poking around in the bootstrap/scss/_variables.scss file trying to sift through a huge list of CSS variables to work out which one does what I want.

```{=html}
<table>
<tbody>
<tr>
<th scope="row">Hello</th>
<td><strong style="background-color:purple; border-radius: 9px; padding: 5px;">text</strong></td>
<td>1</td>
</tr>
</tbody>
</table>
```

title: "Listing Example"
listing:
  contents:
    - "reports/*.qmd"
    - "lab-notes/*reports.qmd"

while true; quarto preview; end

<header id="title-block-header">
$if(title)$<h1 class="title">$title$</h1>$endif$
$if(subtitle)$
<p class="subtitle">$subtitle$</p>
$endif$
$for(author)$
<p class="author">$author$</p>
$endfor$

$if(date)$
<p class="date"><span class="created">$date$ </span>$if(date-modified)$
<span class="modified">— $date-modified$</span>
$endif$</p>
$endif$
<span class="ratings">
    <span class="rating rating-usefulness-${if(usefulness)}${ usefulness  }${else}0${endif}"></span>
    <span class="rating rating-certainty-${if(novelty)}${ novelty  }${else}0${endif}"></span>
    <span class="rating rating-novelty-${if(certainty)}${ certainty  }${else}0${endif}"></span>
    <span class="rating rating-polish-${if(polish)}${ polish  }${else}0${endif}"></span>
</span>
$if(audience)$
<div class="audience">
    <span class="notification-title">Assumed audience:</span>
    <p>$audience$</p>
</div>
$endif$
$if(content-warning)$
<div class="content-warning">
    <span class="notification-title">Content warning:</span>
    <p>$content-warning$</p>
</div>
$endif$
$if(abstract)$
<div class="abstract">
<div class="abstract-title">$abstract-title$</div>
$abstract$
</div>
$endif$
</header>

killall -9 deno

quarto preview --no-navigate

quarto preview --port 8887

quarto preview --port 8887 --no-navigate --no-browser

project:
  type: website
  output-dir: _site
  preview:
    port: 4321
    browser: false
    navigate: false

.navbar {
    // --bs-navbar-color: #050505;
    // --bs-nav-link-color: #050505;
    // --bs-navbar-color: $body-color;
    // --bs-nav-link-color: $body-color;
    font-family: $headings-font-family;
    background-image: $bg-shaded-image, $bg-image;
    background-color: $bg-color;
    color: $body-color;
    .navbar-brand{
        // Cannot fucking work out where the header color gets set to something dumb
        color: $body-color;
    }
    // Trying to eliminate that fucking pale header color die die die
    .navbar-nav .nav-link {
        // color: var(--bs-body-color) !important;
        color: $body-color !important;
    }
}

6 Supporting scripts

The keyword to inject headings into the page is includes, for example, include-in-header or include-after-body.

7 Search

There is a built-in search but it gets unwieldy for a big blog like this.

I use the algolia search, or at least I attempt to. Setup was not obvious. For some reasons using the algolia crawler did not work as it did with blogdown. I needed to manually update the search index, which is not obvious, nor documented

We need the algolia CLI and jq:

brew install algolia/algolia-cli/algolia jq

Now after a render we can do this:

jq --compact-output '.[] | .text |= .[0:9000]' _site/search.json | \
  algolia objects import danmackinlay_quarto -F -

The jq command is necessary to split the search index into separate records, and also to truncate them to fit inside algolia’s 10000 character limit.

project:
  type: website
  output-dir: _site
  resources:
    - keybase.txt
    - "*.bib"
    - notebook/*.yaml
    - post/*.yaml
website:
  title: "The Dan MacKinlay family of variably-well-considered enterprises"
  site-url: https://danmackinlay.name
  favicon: _theme/logo.png
  twitter-card:
    creator: "@dan_mackinlay"
  open-graph: true
  navbar:
    right:
      - text: About
        file: about.qmd
      - text: Currently
        file: notebook/currently.qmd
      - text: Incoming
        file: notebook/incoming.qmd
      - text: Blogroll
        file: notebook/blogroll
      - text: Blog
        file: post.qmd
      - text: Notebook
        file: notebook.qmd
      - text: Everything
        file: everything.qmd
  search:
    algolia:
      index-name: danmackinlay_quarto
      application-id: LNWYJ42WO6
      search-only-api-key: a038347e5450a6426f008faf22c1a4c4
      show-logo: true
format:
  html:
    template-partials:
    - /_theme/metadata.html
    theme:
    - cosmo
    - style.scss
    html-math-method: mathjax
    strip-comments: true
    max-width: 1400px
    code-fold: true
    code-line-numbers: true
toc: true
number-sections: true
execute:
  freeze: true
  cache: true

8 Tips

• store site assets by declaring a resources directory

9 Migrating from blogdown

A few people found it easy.

• Migrating from Hugo to Quarto
• Switching to Quarto from Blogdown includes some red-hot hacks such as simply renaming .html files to .qmd

For my purposes I found it best to script a migration, so I could benefit from all the latest fun features. Here is the one-time migration script I used.

#! /usr/bin/env python
"""
walk tree, replace .Rmd with processed .qmd
"""
from pathlib import Path
import sys
from ruamel.yaml import YAML
import re

yaml = YAML(typ='rt')


def replace_internal_links(input_string):
    # Regex pattern to find internal links with optional leading '/',
    # and an optional fragment identifier
    pattern = r'\[([^\]]+)\]\((\.?\/.*?)(\.html)(#.*?)?\)'
    # Replacement pattern, including the fragment identifier if it exists
    replacement = r'[\1](\2.qmd\4)'

    # Replace the found patterns with the new format
    return re.sub(pattern, replacement, input_string)


def replace_math_delimiters(input_string):
    # Function to determine the replacement based on single-line or multi-line
    def replacement(match):
        text = match.group(1)
        if '\n' in text:
            # Multi-line match
            return f'$$ {text} $$'
        else:
            # Single-line match
            return f'${text}$'

    # Replace math: \( ... \) with $ ... $ or $$ ... $$ depending on single-line or multi-line
    math_pattern = r'\\\((.*?)\\\)'
    input_string = re.sub(math_pattern, replacement, input_string, flags=re.DOTALL)

    # Replace display math: \[ ... \] with $$ ... $$
    display_math_pattern = r'\\\[(.*?)\\\]'
    input_string = re.sub(display_math_pattern, r'$$\1$$', input_string, flags=re.DOTALL)

    return input_string

def read(fname):
    metadata = {}
    outlines = []
    with open(fname, 'r', encoding='utf8') as fp:
        lines = fp.readlines()

    if len(lines) == 0:
        return {}, ""

    if lines[0] == ('---\n'):  # YAML header
        # Load the data we need to parse
        to_parse = []
        for i, line in enumerate(lines[1:]):
            # When we find a terminator (`---` or `...`), stop.
            if line in ('---\n', '...\n'):
                # Do not include the terminator itself.
                break

            # Otherwise, just keep adding the lines to the parseable.
            to_parse.append(line)

        parsed = yaml.load("".join(to_parse))

        for k in parsed:
            name, value = k.lower(), parsed[k]
            metadata[name] = value

    else:
        for i, line in enumerate(lines):
            kv = line.split(':', 1)
            if len(kv) == 2:
                name, value = kv[0].lower(), kv[1].strip()
            else:
                break

    if len(lines) > i+2:
        for line in lines[i+2:]:
            outlines.append(line)
    return metadata, "".join(outlines)


def write(fname, metadata, content):
    with open(fname, 'w', encoding='utf8') as fp:
        fp.write('---\n')
        yaml.dump(
            metadata,
            fp,
        )
        fp.write('---\n')
        fp.write(content)


def massage_one_file(rmdname):
    """
    very minor tweaks to update for Quarto metadata.
    """
    stem = str(rmdname.stem)
    htmlname = rmdname.with_suffix('.html')
    yamlname = rmdname.with_suffix('.yaml')
    bibname = rmdname.with_suffix('.bib')
    is_listing = False
    is_tag = False

    if rmdname.parts[1] == 'tags':
        is_tag = True

    if stem.startswith('_index'):
        # ignore the path part, use the parent dirname as the base of a new file
        pathparts = rmdname.parts[1:-1]
        if len(pathparts):
            if is_tag:
                pathparts = tuple(['_tags', *pathparts[1:]])
            qmdname = Path(*pathparts).with_suffix('.qmd')
        else:
            qmdname = Path('index.qmd')
        is_listing = True
        indexpath = str(qmdname.stem)
    else:
        # Skip the first part and reconstruct the path
        qmdname = Path(*rmdname.parts[1:])

    qmdname = qmdname.with_suffix('.qmd')
    newyamlname = Path(*yamlname.parts[1:])
    newbibname = Path(*bibname.parts[1:])

    metadata, rmdcontent = read(rmdname)
    images = metadata.get('images', [])
    if len(images) > 0:
        metadata['image'] = images[0]
    if 'description' in metadata:
        #rename to 'subtitle'
        metadata['subtitle'] = metadata['description']
        del metadata['description']
    if 'modified' not in metadata and 'date' in metadata:
        metadata['date-modified'] = metadata['date']
    if 'modified' in metadata:
        metadata['date-modified'] = metadata['modified']
        del(metadata['modified'])
    if 'tags' in metadata:
        new_tags = [s.replace("_", " ") for s in metadata['tags']]
        # rename to 'categories'
        metadata['categories'] = new_tags
        del metadata['tags']
    if is_listing and not is_tag:
        metadata['listing'] = {
            'contents': indexpath,
            'feed': True,
        }

    qmdname.parent.mkdir(parents=True, exist_ok=True)
    qmdcontent = replace_internal_links(rmdcontent)
    qmdcontent = replace_math_delimiters(qmdcontent)
    write(qmdname, metadata, qmdcontent)
    print(f"writing {rmdname} to {qmdname}")

    if rmdname.is_file():
        rmdname.unlink()
    if htmlname.is_file():
        htmlname.unlink()
    if yamlname.is_file():
        yamlname.rename(newyamlname)
    if bibname.is_file():
        bibname.rename(newbibname)


def main():
    glb0 = "content/**/*.Rmd"
    #TODO: check for non-relative paths
    paths = Path('').glob(glb0)
    for fname in paths:
        massage_one_file(fname)
    # move the remains per default maybe that works OK
    glb1 = "content/**/*"
    paths = Path('').glob(glb1)
    for fname in paths:
        newfname = Path(*fname.parts[1:])
        newfname.parent.mkdir(parents=True, exist_ok=True)
        if fname.is_file():
            fname.rename(newfname)
            print(f"renamed {fname} to {newfname}")

if __name__ == "__main__":
    main(*sys.argv[1:])