Michael Wolf
Projects and thoughts from Cincinnati San Jose Oakland an undisclosed location in Grass Valley
Home Projects Photos Github Mastodon Nullbrook RSS
Organizing Personal Knowledge Repositories
Last edited - 2019/03/16

This post on Hackernews about Polarized, an online document and note manager, was the first time I read the term "personal knowledge repository." I had long been maintaining systems that served this purpose, but I think this term in particular captures the purpose of something I hadn't quite been able to define in concrete terms. A personal knowledge repository can be a dumping place for thoughts, notes, reflections, interesting content, memories, dreams, bookmarks, or anything worth referring back to later.

Evernote long reigned supreme in the note-organizing space of the smartphone era, but with cheap hosting costs and the proliferation of open source tools, a new generation of note-storage and archiving tools have been published (or, at least, come to my attention). Many of these have the benefit of being open-source and able to be self-hosted, avoiding the perils of centralized services

The Polar article states that the value of a personal knowledge store is derived from the quality of the content hosted within it, the compatibility of the storage format, and the ease of use on a regular basis. The author opens the case of a business that fails losing its wiki. As a hardcore Confluence contributer, this situation of frantically pulling out my lessons learned on my final days has been real. I believe Polar has framed this discussion of a store of valuable, long-term information held in an open-standard information format. However, I disagree with Polar's centralized hosting, Google Analytics tracking and their suggested model of revenue. Thus, I will be discussing what I have (and haven't) found valuable in my note-taking and journaling.

To me, any of the following can qualify as a personal knowledge repository:

Currently I use the following knowledge stores:

Previously I have used:

Presently my knowledge repos are for my own consumption, but it may make sense one day to create read-only versions of them to share publicly like Nikita Voloboev's personal knowledge store. This would help to share what I find interesting in organized way (unlike Twitter) quickly (unlike writing a blog post on each topic). Looking forward, I may set up a connection from this website to those services to share relevant links/notes.

The shift of knowledge and memory into external stores isn't new, obviously. Papyrus, cuneiform, books, statues, tractor-fed printouts, and the almightly 3-ring binder all served to offload the cognitive task of memorization. As an individual utilizing offloaded memory as much as I can, I feel as though my wetware has become shriveled. I can hardly recall the lyrics to a Joyce Manor song let alone the epic poems that minstrels of medieval times memorized. Yet, I firmly believe that my brain, weakened over time by aging and alcohol consumption, is less reliable at retrieving non-critical information than a regex-parsed document.

My Relationship With Personal Knowledge Repositories

I've been an avid journaler since around my elementary school days. The entries were trivial, but stored memories I wished to save for the future. They are incredibly amusing to read (if the handwriting can be deciphered). It's possible that my desire to write down my important memories came from an early realization of how quickly memories condense into shallow snapshots of the past. The whole existential crisis of the inevitable passage of time hit me before I had entered puberty and I took to scrapbooking and storing notes, love letters, trinkets, etc in shoeboxes. At the same time, digital technology allowed me start hoarding photos and gigabytes worth of stupid memes and forum content on my hard drive.

I've spent many days of my life organizing my computer filesystems-- neatly hierarchical by chronology but named in the days before I understood the value of using epoch time. Similar efforts have gone into putting the old notes and doodles from school into a readable, time-ordered format. These collections are mostly useless for knowledge retrieval, but have been invaluable for reminiscing. Looking back might not be the best way to maximize personal utility, but I can't help but love looking back at history on a small scale.

This post about the culture of meme collecting got no small amount of trash talk on Hackernews for being a bit absurd. While the concept of buying and selling memes or valuing how mangled they have gotten during lossy recompression is certainly silly, I feel that there is truth in how these relics of Internet browsing gone by serve as the mementos of my generation. The early internet is a rosy place for many of the first-world millenials. Sites hosting absurd graphic design, cluttered interfaces, and edgy communities fostered by seemingly good-natured tech companies were where I grew up. Albino Black Sheep, Newgrounds, 4chan, DeviantArt and many others I have probably forgotten helped usher me into the collector that I am today. I need to do a thorough analysis of the breakdown of my file system now, but it's not out of the question that I have more downloaded memes and gifs than I have photos from my own life. And it's pretty much been useless. I have yet to crack open the stale folder of reaction images to turn into Telegram stickers. Yet, I'm proud to have some offline copies of internet culture history.

Simultaneous to the meme collecting and scribbled secret journals that I thought anyone actually cared about, I was starting my educational note-taking exercises. Whether it was due to my illegible handwriting or my failure to transition the knowledge into a digital dimension, nearly all of those notes are inaccessible to me. They are rotting away in a converted dresser in a bedroom at my parents' house. I do wonder what my personal knowledge repository would look like if I had managed to start it while I was in high school or something like that. Unfortunately, laptops were essentially banned in class and probably for good reason. Even during enjoyable classes in college, I would find myself inexplicably playing Civilization V. The handwritten notes did serve the purpose of enforcing memorization. I did well in school and I managed to actually derive a love of knowledge from those teachers that didn't spend the whole class lecturing on discipline.

One of the keystones of my note-taking while in school was the daily planner. It was critical in the days before internet-based course-management platforms to make sure you at least knew what homework you had to finish in the minutes before the next class started. While ten years later I could care less about what pages in the biology textbook I needed to read, the scribbles and doodles that I left in these books that I took pretty much everywhere are valuable to me as I age. Most of my schoolwork has been tossed (aside from a few essays and projects) but I've kept these antique places to organize your weekend, share phone numbers, and flirt with girls. These planners lasted longer than the ink on my hand, my ability to access a school digital platform, or any childhood social media banter I didn't intentionally screenshot. It's hard to grok, but at least I own it.

I went to college during the transition from paper assignments to digital ones. This made a dramatic change to how I kept notes. After a few weeks I realized that my handwritten planner was simply marking the dates of upcoming DIY events and I dropped it in favor of Facebook Events. The only thing I was still handwriting was my daily journal and my travel log. Looking back, this was terrible. I didn't realize at the time, but almost all of the notes I took ended up getting sucked into the Blackboard system which I no longer have access to (and I think was actually recently replaced with a new system). I still have the essay Word files and the long Google Doc scratch sheets where my research adviser told me how I was doing things incorrectly. If a class was only Blackboard discussions then essentially all I have left from it is the scrawled handwritten notes I took. I certainly didn't take notes in coding classes.

My phone took on a lot of the notes that would have otherwise fallen into the margins of a planner. I began to use an offline notepad app to store TODOs, lists of things to look for at thrift stores or bookshops. It stored links to projects I wanted to recreate, phone numbers of people I met, mailing addresses, and miscellaneous thoughts. The main issue with this stack-style datastore (aside from the fact that I could quickly lose all of this RAM if I drop my phone in a sewer) was the rapid speed that it would fill up. Urgent things weren't really prioritized and non-urgent things were forgotten near the top of the very long text document.

When I began working in an office, I traded Blackboard for JIRA and I legally couldn't take my code away with me. Thus, I continued my practice of note-taking with renewed vigor. Hand-drawn diagrams of infrastructure, logic flow, and architecture start out neatly but become more scrawled as I transition from intern to programmer. By the time I get in the swing of things at a given job, my notebooks become messes of Jira ticket numbers, circled names of engineers, and contextless phrases that are really just pointers to a space of wetware memory. I maintained wikis aggressively to store issues that I had run into and may meet again some day. Vizio, UML and ASCII charts fit into long technical documents for the team or short notebooks to myself. And then the train would go into a tunnel and Confluence would be uneditable.

Around 2018 I started trying out tools that could handle searching and quick note-taking. This became particularly necessary as I found myself trying to pull up old code snippets or thoughts. Confluence and a written journal weren't cutting it. I initially tried Workflowy, but the cloud model really bothered me. It also didn't store links well. I had been very impressed by how people like Gwern manage to store a near infinite set of links on various topics. Storing them in a hierarchical format seemed like it would quickly get out of hand. Thus, I was able to come across my current digital setup -- phone notes for general todo items and lists, Trillium Notes for work logs and code snippets, and Reminiscence for bookmarks and archiving websites for later. It's been working pretty well since then, but I still keep thorough written journals.

Next Steps


Handwritten Daily Journal/Dream Journal

My daily journal has essentially been ongoing since I was in elementary school, but there have definitely been breaks in the entries. Throughout high school it was more interspersed with fiction, doodles, and embarassing gushing, but as I've gotten older it has been more factual with some color or fun observations that I wanted to remember. I've gone through half a dozen notebooks, generally cheap college-ruled and ring-bound. The most consistent element of the notebook has been my atrocious handwriting. Most likely derived from my bizarre incorrect pen-grip, my handwriting could probably be considered a secret code if it was actually legible to me. Unfortunately, parsing my handwriting requires a substantial amount of effort on my part. In order to try to improve the legibility of my journals, I switched from printing my letters to cursive in high school. Unfortunately after a year, my cursive was just as bad. I can't really say that journaling has improved my memory so this form of note-taking has really benefited me at all. Yet I still do it!

I used to keep a separate dream journal by my bed when I was training myself to lucid dream (the poor man's psychedelic). I would read a few entries before bed and then do my best to brain dump the dream into the journal as soon as I woke up. My success rate for getting at least an outline of the dream recorded was around 20%. Being groggy in the morning my handwriting was extra atrocious. I had some nasty lucid dreams that discouraged me a bit, but I kept up the journal well through college. Ironically, I've had many nights where I record the dream in the extensive detail only to wake up and realize that I was dreaming and I hadn't written anything at all. I may write another post on dreams at some point.

Today my dream journal and daily journal are combined into one notebook and I sometimes write dreams into my phone/Trillium if it's more convenient. This combination journal holds all entries for nights I slept in my bed or somewhere that was no more than a train-ride away. It's taken on the scrapbook aspect of the travel journal where I have made a point to paste in flyers, photos, maps, pressed flowers and business cards. This helps navigate the timeline of the notebook and kick my memories into gear when I can't read the actual words.

Handwritten Travel Journal

I started my travel journal when I finished high school and was about to go on a road trip across the United States to the Montana Rainbow Family Gathering with my friend. I picked out a nice, pseudo-leather-bound notebook and put in timestamped entries with an inky G6 pen at every gas station and each day at the campsite. I've maintained this journal for any trips where I don't sleep in my own bed or a place that would reachable by public transit. This journal was where I began pressing flowers and pasting in maps. I put together some very pretty pages in these journals and treasure these mementos above most of the others. I've since started using my girlfriend's fancy lined notebooks.

Handwritten/Assembled Scrapbook

I had at one point put all of the miscellaneous flyers, receipts, photo booth printouts, and ticket stubs on my wall with tape and thumbtacks but that eventually became unwieldy and, as I got older, a bit more unsightly. So I settled for a scrapbook. These scrapbooks have no writing, but follow a chronological order so it can be pretty easily matched up against the daily journal and travel journal. Every time I get a memento that doesn't quite fit in my smaller journals, it goes in a box that will eventually be sorted into the scrapbook. The scrapbook is intrinsically connected to my relationship with Yvonne. It holds memories that we've made together rather than entries I have made about work or whatever tech project I'm working on. It's pretty sappy.

Handwritten Work Journal

My work notebook follows me pretty much everywhere in the office much like my laptop. It's full of scrawled tasks and diagrams that many engineers will identify with whiteboards. When I started in the working world, I kept some hardcore notes. Pretty much anything one of my mentors said ended up in the journal. Eventually as I began working on projects, the notebooks actually began looking competant. Flow and logic diagrams, system architecture, and pseudocode lived next to doodles and more JIRA ticket numbers. At the time, I felt that the notebook served as a good representation of how I think and would be something I'd almost want to put on my resume or something. However, my notebooks anymore look like the wild jottings of a seance or a hostage victim. I suppose that is the way of operations.

Handwritten Marginalia

Throughout my elementary and high school literature classes, teachers encouraged (and even graded) students to tear through classic novels with a highlighter or a pen-- pulling out key phrases and ideas that could be shared with the class. This didn't occur to me to be a particularly useful exercise outside of ensuring that the kids were actually flipping through the book. However, as I began my honest love of reading in college, I found that marginalia was a useful way of engaging with books, making them personal, and probably preventing them from ever being valuable to anyone else. While I'm not one of the sickos that would mark up a library book, my own books are covered in loads of snarky comments or circled paragraphs. I hope that they come in handy as I put together my book reviews. I've found the discussion of historical figures' interactions with marginalia to be particularly inspiring. Jane Austen's World, and The New Yorker both have bits of praise for the interesting knowledge that can be gleaned from writing in books.

Snow Crash
Pharmakopoeia
Logic Magazine #6
Smartphone Offline Notes App

I've been using the Simple Notes app on Android ( Play Store, FDroid, Source) from Simple Mobile Tools and I couldn't recommend it enough. It's a no-nonsense space for jotting notes and has a solid homescreen widget. It doesn't need an internet connection and it's extremely snappy. In the notes system I have separate txt files for TODO, grocery lists, mailing addresses, etc. It's incredibly convenient, but I'd be pretty fucked if I dropped my phone in the toilet. The notes I store here aren't worth backing up.

Trilium Notes

Trilium (Source) has become the place that I store digital notes that are worth backing up or need to be searchable. Trillium is a self-hosted node-based notes server with a pretty nice web interface. It stores notes in markdown and allows for hierarchical structuring of notes. The server is simple enough that I have a simple rsync set up to back it up. The software supports syncing from a local instance to a server instance which I assume would support offline work, but I haven't found that necessary so far. Many thanks to zadam.

Trilium is now home to much of my work journal, learning notes, observation journal of public places, dream journal, and random tidbits. It's an intuitive and quick application that just works. Plus I don't have to worry about being unable to read my handwriting. Time will tell if the search function scales up with the amount of content I'm likely going to import.

Reminiscence Bookmark System

Reminiscence (Source) is part bookmark manager, part way-back-machine. It allows for storing links as tagged and searchable objects within folders as well as saving copies of the websites using wkhtmltopdf or as plain html. Bookmarked sites can then be annotated with a summary. Since my instance of Reminiscence is hosted on a tiny Digital Ocean droplet I haven't been offlining the websites for sake of disk space. However, I do plan to go back and utilizing some of Archive Team's tools like wpull and take larger backup chunks of the Internet.

Presently, my instance of Reminiscence is behind a login, but I may open it up to be read-only at some point in order to share the links I've collected. It's not like there's anything I'm ashamed to have saved, right?

Digital Filesystem of Photos/Videos/Notes

My filesystem is probably my most prized possession. It has a hot backup and a cold backup. It was the first thing I ever had in a RAID array. It consists of every photo/video I've taken since 2007 but has content from before I organized things well. It's incredibly embarassing to look through some times, but it has become a treasure trove of reminiscing with friends-- especially when I don't use social media to catalog my adventures. At the time of this writing it is 217,423 items totalling 443 GBs. It's not a massive number but this is because it is entirely photos or videos I have taken with a digital camera, camcorder, or phone (my film photography is currently 1500 items totally 6.5 GBs). Something inside me compels me to backup this content religiously while my peers throw away their photos on Snapchat stories and purged iCloud accounts. I can't really say that this archive has done me any good but when I see that I haven't taken any pictures in awhile, I encourage myself to go out and do so. This Too Shall Pass.

While less valuable then my personal photos, I also continue to store a decent chunk of Internet media. My collection of 500 GB of music from What.CD (RIP) was destroyed by a power surge in 2015, but most of my photos and gifs from the late-aughts, early-teens Internet remain. I have about 10 GB of memes from forums that are now essentially unused and about a thousands photos from Photobucket's recent uploads that I used to use to fill a Tumblr blog of images out of context. Nearly all of these downloads were from manual right-clicks during the hours I spent on the wastoid corners of the Internet when I was still living at home. Luckily I spend my time pursuing much more worthwhile endeavors like taking notes about note-taking, refreshing Hacker News, and putting off cool projects.


Conclusion

My thoughts and experience with notes could have probably been summarized with greater brevity, but my point is this. Go, take notes. Take hella notes. The more legible and quickly searchable they are, the better they will age and greater benefit they will bring to you. Finally, don't lock yourself in to any storage system. Own your personal IP. Share it as much as you want, but make sure it's always there when you need it.