Coding Notebook

What is This?

This is a public-facing notebook about what I’m doing, what I’m struggling to do, and what mistakes I’ve been making.

Week 1

This week (May 29 to June 4), I set up a preliminary animint2 reference website.

Learned

• Learned more about the pkgdown R package:
  • Began looking into the pkgdown documentation.
  • Webpages seemed to be generated from .rmd files, so I was curious if the package also worked the same way for .qmd files. The answer is no.
  • Looked up why pkgdown adds the docs/ directory into .gitignore. But it’s still unclear to me.
  • Noticed that the intro to pkgdown page accidentally uses the word “setting” twice in a row.
• Learned more about git:
  • Learned of gitk, which I’ve never used before.
  • Learned about managing my git credentials in usethis.
• Looked up some tips on how to write code comments better. Tips from: Antonin Januska and Ellen Spertus.
• Learned that the string GITHUB_PAT can be found in the .github/workflows directory.
• Learned about documentation managers. There’s: Zeal and DevDocs.
• Learned that there are, like, eight test suites. And they automatically run when you make a pull request to the animint2 repository. That’s nice. I’m also learning that the tests take a long time to run—about 45 minutes.
• Turns out that ripgrep-all doesn’t look through YAML files. I need something like yq instead. Why?

Created

• Created README.md from the README.org file via the magic of pandoc.
• Created a preliminary website and documented how I generated it:
  • For the documentation and code, see website_setup.Rmd. Currently have information about building the website.
  • Put the site up online.
• Accidentally made a pull request in tdhock/animint2. Should’ve been ampurr/animint2 instead. I closed it—be more careful.
• Made a pull request in ampurr/animint2 but did not initially succeed in closing it due to a test failing. Later, I pushed it anyway, which then threw the website online.
• Successfully set up the website. But there are some errors. Most noticeably: it includes some functions that don’t seem to be part of animint2. Will need to investigate further.

Confused

One of the tests—“Test Suite compiler”—failed. Here’s the error message:

Code

══ Failed tests ════════════════════════════════════════════════════════════════
── Error ('test-compiler-gist.R:4:3'): animint2gist() returns an object of class 'gist' ──
Error: Error: In non-interactive environments, please set GITHUB_PAT env to a GitHub access token (https://help.github.com/articles/creating-an-access-token-for-command-line-use)
Backtrace:
    ▆
 1. └─animint2::animint2gist(list(p = qplot(1:10)), browse = FALSE) at test-compiler-gist.R:4:2
 2.   └─gistr::gist_create(...)
 3.     ├─gistr:::gist_POST(...)
 4.     │ └─gistr:::cVERB(...)
 5.     │   └─crul::HttpClient$new(url, headers = c(auth, headers), opts = list(...))
 6.     │     └─crul (local) initialize(...)
 7.     └─gistr::gist_auth()

[ FAIL 1 | WARN 0 | SKIP 5 | PASS 756 ]
Error: Error: Test failures
Execution halted
Error: Process completed with exit code 1.

This is confusing, since I thought GITHUB_PAT was already set with a GitHub Access Token. I was also unaware of the large array of tests that activate when you try to push something. Quite frankly, I’m not sure what they do or what they prevent. It feels different than using testthat.

I also don’t understand the GitHub workflow stuff. What exactly is GitHub doing?

Week 2

This week (June 5 to June 11), I tried to fix issue 89. I did not succeed.

Learned

• Learned that the roxygen2 package generates both the .Rd files and part of the DESCRIPTION for packages.
• Learned that devtools AUR package has so so so many dependencies.
• This is real basic, but I re-learned that bash foo.sh runs shell scripts. I haven’t run one since September of 2022.
• Looked into Writing R Extensions, which I was not previously aware existed:
  • Read the DESCRIPTION file portion.
  • Learned about RFC 2822, a syntax standard for text messages in email. FastMail mentions that RFC 5322 makes RFC 2822 obsolete, but Writing requests emails be written in RFC 2822 formatting.
• When using Git Bash—i.e. MinGW—and cd, I need to swap the backwards slashes for forward slashes. Oddly, the backwards slashes work for some directories but not for others. It took me a while to realize this while debugging.
• Figured out that the borders() function is part of fortify-map.r. That’s why it disappeared when I deleted that R file.
• Learning about bash.
• Learned that all the tidyverse data tidiers will eventually be retired. This includes both ggplot2::fortify() and broom::tidy().

Created

Nothing of value. I created several branches to try and fix the issue and didn’t succeed.

• One branch focused on removing the fortify functions in animint2. This caused several tests to fail. Later, Toby told me that map_data is used in several tests and relies on the fortify functions.
• As part of a debugging process, one branch was a copy of the main branch with an additional text document. It was meant to test the GitHub tests. It passed the tests.
• Another branch was meant to debug the branch where I removed the fortify functions.

Failed

My debugging process didn’t succeed. Or, at the very least, it didn’t succeed in time.

Parts of it were headed in the right direction. After some experimentation, I confirmed that the problem stemmed from my changes to the repository (the removal of the fortify functions) and not from either build.sh or the GitHub tests. But my debugging was focused on the following error message:

Error in FUN(X[[i]], ...): numeric ‘envir’ arg not of length one.

This error message comes from the ggplot2 side of animint2’s code. In some sense, I was looking in the right direction—the fortify functions are from the ggplot2 side of things. But I was focused on figuring out substitutes for fortify and failed to notice that the problem was with map_data.

Confused

For some reason, build.sh always fails to generate the \(\LaTeX\) document correctly, mentioning some problem with the .Rd files. Then I think it runs the process again, and the problem disappears. Why?

Why are the tidyverse data tidiers being retired? I can’t find any info about that.

yay -S r-plyr r-reshape2 isn’t working and requires “manual intervention.” Why? Luckily, this isn’t important—build.sh runs perfectly fine on Git Bash and therefore I don’t actually need to figure out why the yay helper is acting this way.

Week 3

This week (June 12 to June 18), I complained a lot. I chatted with Fei, who said that she would take care of issue 90. Thank you, Fei!

Learned

• First meeting with Toby!
  • animint2 doesn’t compute—all computations are done beforehand, and then animint2 displays that as an interactive data visualization.
  • Also learned that capital letters go before lowercase letters in terms of alphabetical order. At least according to R.
  • build.sh, and some of the GitHub Actions, are meant to emulate CRAN checks.
  • All else being equal, prefer small changes over large changes.
• Looked up selenium documentation in order to figure out if a test case for issue 90 was possible.
• Preliminary research on alt-text for ggplot2, graphs, and data visualizations:
  • N. Elmqvist. Data Visualization for the Blind. In Interactions. 2023.
  • Office for National Statistics. Alt Text: A Guide to Writing About Statistics.
  • C. Jung and others. Communicating Visualizations without Visuals. In ArXiv. 2021.
• Learned about rtools, which is necessary to make R packages in Windows:
  • Learned that rtools is not an R package, but must be installed separately.
  • Found some rtools documentation.
  • Toby sent a link that mentioned rtools needs MikTeX and Inconsolata.
• Discovered the rmini package, which I’ve been using for testing purposes.
• The R Packages book has an appendix about R CMD checks, including bug fix suggestions for \(\LaTeX\) errors.
• Took a look at Chapter 12: Packages in R for Fledglings, but it wasn’t helpful.
• There’s a Stack Overflow question about \(\LaTeX\) errors in R packages.

Created

• Made a comment in issue 90. Fei fixed Big Bug, but Little Bug is left.
• Made a new issue: issue 96, which is about some \(\LaTeX\) errors that seem exclusive to me.

Week 4

This week (June 19 to June 25), I tried to set up the reference website again—this time in Toby’s repository instead of my fork. This didn’t work, so I started tinkering with my fork again.

Learned

• Here are the GitHub Docs. Look here when you have a question about GitHub.
• Looked up how to get permission to change GitHub Page settings in the animint2 repository. One Stack Overflow post suggested obtaining admin rights. Instead, Toby created an organization for repository and gave us permissions that way.
• Looked up info about the GitHub Pages API.
• Refamiliarized myself with aspects of git that I’ve never used before:
  • On detached HEADs: Stack Overflow: How Do I Fix a Detached HEAD? and How to Reattach a Detached HEAD.
  • On undoing merges in git: Stack Overflow: How Do I Revert All Local Changes? and How to Undo a Merge. Here’s a blog post on undoing merges with git reset.
  • GitHub: Reverting a Pull Request.
• Learned more about GitHub’s personal access tokens:
  • GitHub Docs: Managing Your PATs.
  • GitHub Docs: Caching Your GitHub credentials in Git.
  • Stack Overflow: Where to Store My GitHub PAT?.
• Learned a little more about GitHub Actions:
  • You can choose which GitHub Actions to re-run. Nice! When a test fails, I can just rerun that test instead of waiting 45 minutes for everything to run again.
  • GitHub Docs: About GITHUB_TOKEN.
  • A blog post about the GITHUB_TOKEN in GitHub Actions.
  • There’s documentation for usethis::github_token.
• There’s an error that I often get: In non-interactive environments, please set GITHUB_PAT env to a GitHub access token. Where does it come from? After doing a GitHub code search, I figured out that it’s from the gistr R package.

Created

A mess. I accidentally merged my changes from my fork of animint2 into the original animint2 repository. Whoops.

Week 5

This week (June 26 to July 2), I’m setting the website up again. Yay! Toby said that he wants me to replace the README entirely, so that’s happening, too.

Learned

• You know issue 96? The one about \(\LaTeX\) errors? I tried installing Inconsolata, but it didn’t fix the issue. My Inconsolata font is the variable one from Google Fonts.
• I wanted to know if it was possible to manually run a GitHub Action thing—that way, I would know if I’m failing a test before I submit a pull request. Turns out the answer is yes, but only if I have the workflow_dispatch trigger. The workflow in animint2 doesn’t have that trigger, though.
• Learned that the Windows 11 snipping tool can record videos.
• Learned how to trim videos with ffmpeg.
• I dunno what it does, but there’s this R Universe thing. You can search for animint2 and get information about it.
• Learned that data() will generate a list of all of R’s built-in datasets.
• Learned that I can generate GitHub-flavored Markdown (GFM) with Quarto. Here are Quarto’s options for GFM.
• R and ggplot2 have built-in datasets, and I found the codebooks! These codebooks are for R. And these are for ggplot2.

Created

• Toby created an organization, animint, for the animint2 repository. I did some things with it:
  • I changed the profile picture to a dittered Suki. Improving the world one cat pic at a time.
  • I changed the settings so that we can control things with our personal access tokens (PATs). If you want, go to Settings > Third-party Access > Personal Access Tokens to change it back.
  • I generated a PAT for animint2.
• Converted README.md to README.qmd, which will let me do R computations for the website homepage.
• Added the “Differences with other packages” section from README.org to the animint2 repo wiki.

In Progress

• I’m thoroughly updating and cleaning up the website_setup.Rmd document.
• I’m editing README.qmd to be clearer and more immediately useful. Later, I’ll convert it to README.md for pkgdown to read.

Week 6

Hoping to finish up the website this week (July 3 to July 9).¹ Weird bugs exclusive to me and GitHub-specific problems… I’m behind on my timeline for reasons I didn’t anticipate. Good learning experience, though I’m learning about things I didn’t anticipated.

¹ Editor’s note: this didn’t happen.

Learned

• Unsurprising, but I learned that you can’t have JavaScript in your GitHub README. This is generally untrue: it’s possible to embed Javascript in Markdown.
• For Windows, it’s easy to change the localhost default browser.
• Remember: in RStudio, use Ctrl + Shift + C to comment out multiple lines.
• Load a built-in dataset with data() and use View() to see a dataset more fully.
• Looked up datasets. Found one about school shootings by the Washington Post. Also found the worldPop dataset in animint2.
• Looked up the CRAN List View on probability distributions.
• For subsetting, see Advanced R: Subsetting Data and this post on deleting rows.
• Some useful links for when I’m simulating data:
  • Use rep() to repeat a value some amount of times. Link. Another link. Third link that suggests other functions.
  • Use data.frame(vector1, vector2) to concatenate vectors into columns. Link.

Created

• Created vignettes/simple-examples.Rmd to house the usage examples for the README document. Why? Cuz GitHub doesn’t allow JavaScript in its READMEs.

Other Notes

• When using animint::labs(), subtitles, tags, and captions don’t work. ggplot2 page for the function is here.

Week 7

This week (July 10 to July 16), I’ve continued work on the quick start guide.

Learned

• Re-reading chapter 3 and chapter 4 of the Animint2 Manual.
• Cosma Shalizi’s notes on R Markdown. Read this before—it’s a useful reference.
• R Helpful Codes has a bunch of little R resources and tutorials. Advanced R mentions order().
• There’s a Stack Overflow post with R Resources, including an R reference card that I’ve never seen before.
• Some more useful notes for data simulation:
  • sort() lets us sort our data. order() works, too, and in a more sophisticated way.
  • cummax lets you obtain the largest scalars in your vector. It is related to cumsum()
  • Use rbind() |> c() to weave multiple vectors together.
  • It’s pretty easy to sample from multiple vectors and then concatenate them.
• James Murray has a collection of R tutorials, including one on the grammar of graphics.
• Use [link](#foo-bar) to link to the foo-bar section in a Markdown document. Link.

Created

• Massaged simulated data until they were useful and looked somewhat real.
• Continued work with simple-examples.Rmd, which was renamed quick-start.Rmd. The simple examples vignette has transformed into a larger quick start guide.

Week 8

This week (July 17 to July 23), I finally finished the quick start guide. And the README, too. I also worked a little on website_setup.Rmd and _pkgdown.yml.

Learned

• Erik Bernhardsson has an analysis for why software projects take longer than expected. For obvious reasons, it’s something I’ve been thinking about regarding the reference website project.
• Learned (or re-learned) about the signif() function, which is useful for rounding and cutting off overly-precise numbers.
• Data Visualization: A Practical Introduction has a section on broom in Chapter 6. Useful, since it’s unclear to me what data tidiers do.
• There are ways to place plots side by side in ggplot2, but those methods don’t work in animint2. ggplot2: Elegant Graphics for Data Analysis link. Stack Overflow link.
• You can remove legends and axes in ggplot2. These solutions throw up errors when used with animint2.
• In Notepad++:
  • Edit > Blank Operations offers you the option to delete trailing whitespaces.
  • AltShift throws you into column mode. Good for mass editing.
• Comments are the same in both R and YAML. Use the # sign.

Created

• Finished drafts for quick-start.Rmd, README.qmd, and README.md. Yay!

In Progress

• Worked on _pkgdown.yml.

Other Notes

• animint() fails to render ggplots that uses geom_smooth().

Week 9

This week (July 24 to July 30), I continued working on pkgdown.yml.

Learned

• The ggplot2 repository has a useful _pkgdown.yml file. I used it as a model for animint2’s file.
• Turns out that there are lots of whitespace characters in Unicode.
• I got a parser error in my _pkgdown.yml file, so I looked it up:
  • Looked through the YAML specs. In YAML, it turns out that whitespace is syntactically significant in some places and ignored in others.
  • This Stack Overflow response notes there are at least 9 ways to write multi-line strings in YAML and comments: “YAML is crazy.”
• I tried to download the YAML specs as a PDF so it would be easier to read off the computer. This led to a wild adventure:
  • The YAML repo uses a Markdown file that does not play well with pandoc.
  • I’d forgotten that R Markdown and Quarto create PDFs differently. In the former, use output: pdf_document or the render function.
  • To convert multiple images in Imagemagick, use magick mogrify.
• I’d forgotten why _pkgdown.yml uses bootstrap: 5. It’s cuz some sites might break from the upgrade, so we need to specify that we wanna use Bootstrap 5.

Created

• Finished the pkgdown.yml draft.
  • There was a parser error, but I fixed it. Turns out it was a whitespace error.
  • But now there’s a couple of new errors.²

² Editor’s note: this was the result of three misnamed topics. They’ve since been changed.

Week 10

This week (July 31 to August 6), I finished _pkgdown.yml.

Learned

• incomplete final line warning message is the result of
• Topics filed under title: internal may show up anyway. This is the result of strangely specific syntax requirements.
• Using an SVG workaround, it’s possible to use CSS in a GitHub README.
• In Notepad++, you can use the CTRLJ shortcut to join lines and remove line breaks.
• There’s an odd warning message when you build or check the site with pkgdown: incomplete final line found on './_pkgdown.yml'. There’s a bug report about it. It’s a YAML thing, but I’m not sure why.
  • Just gonna ignore it for now.

Created

• Finished pkgdown.yml! Woot woot!
  • Reorganized all the topics.
  • Wrote up descriptions for the titles.

Week 11

This week (August 7 to August 13), I put a mostly-finished website online! Woot woot!

Learned

• Fun with git:
  • .gitignore comments use #. Just like R and YAML.
  • You need to merge all files before you can revert anything.
  • On GitHub, you can re-publish a website via the Actions tab.
  • On GitHub, GitHub Pages automatically deploys a pages-build-deployment workflow.
  • You can manage GitHub credentials via the usethis package.
    
• Notepad++ stopped working. Removing the plugins folder resolved the issue, though it causes other problems. Probably something wrong with one of the plugins.
• Reviewed Max Kohler’s advice about alt text.
• Advanced R has a chapter called Debugging.
• Some CSS and HTML links:
  • You can select multiple classes with the :not() psuedo-class.
  • HTML tags can have multiple classes. They’re separated with spaces.

Created

• Did lots of website stuff:
  • Removed docs from the .gitignore website.
  • Finally threw the official animint2 site online. Here it is.
  • Made minor changes to the alt text.
  • Fixed an alt text–figure caption duplication error by using <img src="source" alt="alt text"> instead of ![alt text](source).
  • Moved world_bank_screencast.gif to man/figures. It’s a pkgdown convention.
  • Created pkgdown/extra.css and wrote some CSS to make changes to the website. It’s a pkgdown convention.
• Made a PR to begin closing issue 100.
  • It’s about how some of the datasets lack either a source (2) or a codebook (19). Or both (19).
  • Added a bunch of TODOs in the .Rd files in man/.
  • Added a todo.md file briefly explaining the TODO nomenclature.

Other Notes

• There seems to be a minor accessibility bug with pkgdown. When you use ![alt text](source) and render it as an HTML file, it takes the alt text and renders it twice: as the alt text and also as the figure caption.

Week 12

This week (August 14 to August 20), I worked on improving the website, mostly with CSS.

Learned

• I was previously unaware of CSS’s letter-spacing property.
• pkgdown can’t use Quarto cuz CRAN doesn’t have Quarto installed. Didn’t document this properly till now.
• Resources that I’ve looked at to try and solve the JavaScript problem of animints not appearing in the quick guide. I should note that looking into this problem has been somewhat confusing.
  • Toby told me to look at the z_knitr.R file, just in case it’s something on our end.
  • This 2021 pkgdown issue mentions something about live graphs no longer working. But that issue has been fixed, hasn’t it?
  • By default, JavaScript uses the default MIME type text/javascript.
  • How to Include JavaScript files in R Markdown files.
  • Hello JS in R Markdown.
  • How to Add Extra Files to pkgdown Website.
  • Support for custom CSS and JS was added in 2017. Link.
  • Using RGL in pkgdown Websites.
  • Doesn’t matter—it fixed itself.
• Learned and reviewed some CSS stuff:
  • There’s a difference between child and descendant combinators.
• The Windows version of diff is fc.exe. Link.
• There’s an R package called htmlwidgets. Seems cool.
• There were a lot of resources I consulted when designing the code annotations.
  • Remember: use element#id when selecting IDs.
  • Controlling the gap between the list marker and whatever’s after it is a real pain in the rear. See this CSS Tricks link for an analysis of this problem. See this link or this link on what to do. I just used <span> a bunch of times and then moved those. Thoroughly inelegant, but it works.
  • MDN on <dl>.
  • MDN on list-style-position, which isn’t helpful when you’ve changed things with list-style: none.
  • MDN on accessibility concerns re: list-style: none. Specifically: Safari will stop recognizing the list as a list.
  • Consulted MDN on border when designing the code annotations.
  • To generating capsule- or stadium-shaped objects, apply a super big border-radius. Link. Second link.
  • ::marker only accepts a limited number of CSS properties. MDN link. CSS Tricks link.
  • Because of the limitations above, you need to use workarounds. For example: manipulating different pseudo-elements, like ::before. Second link with a similar solution. Another example, this time focused on borders. See also this link from CSS Tricks.
  • MDN on ::before.
  • MDN on CSS counters, which are useful when hacking CSS lists.

Created

• Fei reported a bug under issue 102. I tested it.
• Finally got the animints working in the quick start guide! I’ve moved the guide to its own section on the navigation bar.
• Some other website changes:
  • Switched the font from system-ui to Roboto Condensed and Roboto Mono.
  • Changed the link color to match the link color on Toby’s website.
  • Increased spacing between each heading.
  • Generated code annotations from code comments cuz I don’t have direct access to the HTML. This was hard!

Confused

Okay. Debugging notes. The problem is that the website won’t render our animints. This seems to be the result of two issues:

• pkgdown renders the animints in the vignettes/ directory instead of docs/articles/.
• Having the animints in docs/articles/ doesn’t appear to do anything.

My guess is that it’s a pkgdown thing. Quarto renders it, so the origin of the problem’s not animint2. The animint2 Manual renders it, too. So it’s not anything to do with R Markdown.

The JavaScript output looks the same as well. It’s a bunch of <script type="text/javascript" src="animint/directories"> for the first codeblock, followed by <script>var chunk_name = new animint("#chunk_name", "chunk_name/plot.json");</script> for any future codeblocks. So there’s nothing wrong with the code being generated.

Maybe the code needs to be generated in the docs/ directory instead of vignettes/? But why? I used fc.exe to look at the files and obviously there’s no difference between them.

Oddly, the bug fixed itself. I just renamed the file and now the animints work. Bizarre.

Week 13

Work product submission week (August 21 to August 28)! 🐈

Learned

• Using R and the magick package, you can convert images to waveforms. This has nothing to do with animint2. But I just thought it was neat.
• Mermaid Documentation for flowcharts.
• Reviewed YAML options in Quarto HTML documents.
• R Markdown does give some control over alt text and captions. You can use {r fig-alt: "foobar"}. Link. Other link. But this obviously doesn’t work for images not generated by a code chunk. It also doesn’t work for animints.
• Fun fact: in R Markdown, it’s possible to include line numbers in code chunks.
• Some links on accessibility in data visualizations. Also, links re: alt text in SVGs and adding alt text to JavaScript.
• Looked up knitr code chunk options here.
• Footnotes weren’t working on the animint2 reference site. Some places that I looked up:
  • The pkgdown output rendering page. Oddly, their footnotes seem to work. Ours didn’t.
  • pkgdown uses Bootstrap popovers for their footnotes. The popovers apparently uses an external library, popper.
  • Bootstrap has this thing called scrollspy that throws up errors in all the browsers I use.
• Some CSS reminders:
  • a {text-decoration: none;} removes underlines from links.
  • Use <sup><a href="#footnote-num">footnote-num</a></sup> and <li id="footnote-num"> to make footnotes. Possible link.
• There’s a weird error I sometimes got: Don't know how to add o to a plot. This was obviously unhelpful. I think it has something to do with ggobjects.
• You can set the working directory for R code chunks. Unfortunately, this doesn’t seem to work for animints while using pkgdown.
• How to use dev tools for JavaScript.
• R Packages has a chapter on pkgdown websites.

Created

• Finished website maintenance document and renamed the file readme_website.md.
• Converted NEWS to NEWS.md and edited it so that pkgdown could read it and render it as the changelog.
• Toby suggested major changes to the quick start guide, which I implemented. He suggested other major changes that I still need to implement.

Additional Work

This section tracks work I’ve done immediately after GSOC ended—August 29th and beyond.

Learned

• Investigated the scrolling bug:
  • Brief notes from Firefox on debugging scrollable overflow.
  • overflow MDN page and overflow CSS Tricks page.
  • zoom MDN page. Weird CSS property supported by Chrome but not Firefox.
  • MDN pages for: font-smooth, display, and height.
  • Used both the World Bank animint and the gsoc-test repository to experiment.
  • Problems with SVG animations in Firefox. Seemed vaguely relevant.
• Creating the aside:
  • aside MDN page.
  • I looked into browser detection so it’d only show for Firefox users. CSS Tricks link. But I learned that this feature is brittle and generally a bad idea.
• A Guide To CSS Support In Browsers, with some notes on browser bugs.
• :target is useful for targeting parts of a page that have been linked to.

Created

• Worked on an <aside> to let Firefox users know about the scrolling bug.
• Tried to work on issue 90. I thought I had a solid lead re: the zoom property, but that hasn’t worked out so far.
• Swapped in README.md for README.org for .Rbuildignore.