index.md 94KB
title: Seeking the Productive Life: Some Details of My Personal Infrastructure
url: https://blog.stephenwolfram.com/2019/02/seeking-the-productive-life-some-details-of-my-personal-infrastructure/
hash_url: b575f1ff37
The Pursuit of Productivity
I’m a person who’s only satisfied if I feel I’m being productive. I like figuring things out. I like making things. And I want to do as much of that as I can. And part of being able to do that is to have the best personal infrastructure I can. Over the years I’ve been steadily accumulating and implementing “personal infrastructure hacks” for myself. Some of them are, yes, quite nerdy. But they certainly help me be productive. And maybe in time more and more of them will become mainstream, as a few already have.
Now, of course, one giant “productivity hack” that I’ve been building for the world for a very long time is the whole technology stack around the Wolfram Language. And for me personally, another huge “productivity hack” is my company, which I started more than 32 years ago. Yes, it could (and should) be larger, and have more commercial reach. But as a nicely organized private company with about 800 people it’s an awfully efficient machine for turning ideas into real things, and for leveraging what skills I have to greatly amplify my personal productivity.
I could talk about how I lead my life, and how I like to balance doing leadership, doing creative work, interacting with people, and doing things that let me learn. I could talk about how I try to set things up so that what I’ve already built doesn’t keep me so busy I can’t start anything new. But instead what I’m going to focus on here is my more practical personal infrastructure: the technology and other things that help me live and work better, feel less busy, and be more productive every day.
At an intellectual level, the key to building this infrastructure is to structure, streamline and automate everything as much as possible—while recognizing both what’s realistic with current technology, and what fits with me personally. In many ways, it’s a good, practical exercise in computational thinking, and, yes, it’s a good application of some of the tools and ideas that I’ve spent so long building. Much of it can probably be helpful to lots of other people too; some of it is pretty specific to my personality, my situation and my patterns of activity.
My Daily Life
To explain my personal infrastructure, I first have to say a bit about my daily life. Something that often surprises people is that for 28 years Iâve been a remote CEO. Iâm about as hands-on a CEO as they come. But Iâm only physically “in the office” a few times a year. Mostly Iâm just at home, interacting with the company with great intensity—but purely through modern virtual means:
I’m one of those CEOs who actually does a lot of stuff myself, as well as managing other people to do things. Being a remote CEO helps me achieve that, and stay focused. And partly following my example, our company has evolved a very distributed culture, with people working scattered all over the world (it’s all about being productive, rather than about “showing up”):
At my desk, though, my basic view of all this is just:
Itâs always set up the same way. On the right is my main “public display” monitor that Iâll be screensharing most of the day with people Iâm talking to. On the left is my secondary “private display” monitor thatâs got my email and messages and other things that arenât directly relevant to the meetings Iâm doing.
For the past year or so, I’ve been livestreaming many of our software design meetings—and there are now 250 hours of archived screensharing, all from that right monitor of mine.
Particularly since I’m at my desk much of each day, I’ve tried to optimize its ergonomics. The keyboard is at the right height for optimal typing. The monitors are at a height that—especially given my “computer distance” multifocal glasses—forces my head to be in a good position when I look at them, and not hunched over. I still use a “roll-around” mouse (on the left, since I’m left-handed)—because at least according to my latest measurements I’m still faster with that than with any other pointing technology.
At the touch of a button, my desk goes to standing height:
But while standing may be better than sitting, I like to at least start my day with something more active, and for more than a decade Iâve been making sure to walk for a couple of hours every morning. But how can I be productive while Iâm walking? Well, nearly 15 years ago (i.e. long before it was popular!) I set up a treadmill with a computer in the room next to my office:
The biomechanics werenât too hard to work out. I found out that by putting a gel strip at the correct pivot point under my wrists (and putting the mouse on a platform) I can comfortably type while Iâm walking. I typically use a 5% incline and go at 2 mph—and Iâm at least fit enough that I donât think anyone can tell Iâm walking while Iâm talking in a meeting. (And, yes, I try to get potentially frustrating meetings scheduled during my walking time, so if I do in fact get frustrated I can just “walk it off” by making the treadmill go a little faster.)
For many years I’ve kept all kinds of personal analytics data on myself, and for the past couple of years this has included continuous heart-rate data. Early last summer I noticed that for a couple of weeks my resting heart rate had noticeably gone down. At first I thought it was just because I happened to be systematically doing something I liked then. But later in the summer, it happened again. And then I realized: those were times when I wasn’t walking inside on a treadmill; instead (for different reasons) I was walking outside.
For many years my wife had been extolling the virtues of spending time outside. But it had never really seemed practical for me. Yes, I could talk on the phone (or, in rare cases, actually talk to someone I was walking with). Or I could be walking with a tablet, perhaps watching someone else screensharing—as I did, rather unstylishly, for a week late last summer during my version of a vacation:
I’d actually been thinking about walking and working for a long time. Twenty years ago I imagined doing it with an augmented reality display and a one-handed (chorded) keyboard. But the technology didn’t arrive, and I wasn’t even sure the ergonomics would work out (would it make me motion sick, for example?).
But then, last spring, I was at a fancy tech event, and I happened to be just out of the frame of a photo op that involved Jeff Bezos walking with a robotic dog. I wasn’t personally so excited about the robotic dog. But what really interested me was the person walking out of the frame on the other side, intently controlling the dog—using a laptop that he had strapped on in front of him as if he were selling popcorn.
Could one actually work like this, typing and everything? After my “heart-rate discovery” I decided I had to try it. I thought I’d have to build something myself, but actually one can just buy “walking desks”, and so I did. And after minor modifications, I discovered that I could walk and type perfectly well with it, even for a couple of hours. I was embarrassed I hadn’t figured out such a simple solution 20 years ago. But starting last fall—whenever the weather’s been good—I’ve tried to spend a couple of hours of each day walking outside like this:
And even when Iâm intently concentrating on my computer, itâs somehow nice to be outside—and, yes, it seems to have made my resting heart rate go down. And I seem to have enough peripheral vision—or perhaps Iâve just been walking in “simple enough” environments—that I havenât tripped even when Iâm not consciously paying attention. No doubt it helps that I havenât mostly been walking in public places, so there arenât other people around. Of course, that also means that I havenât had the opportunity to get the kind of curious stares I did in 1987 when I first walked down a city street talking on a shoe-sized cellphone….
My Desk Environment
Iâve had the same big wooden desk for 25 years. And needless to say, I had it constructed with some special features. One of my theories of personal organization is that any flat surface represents a potential “stagnation point” that will tend to accumulate piles of stuff—and the best way to avoid such piles is just to avoid having permanent flat surfaces. But one inevitably needs some flat surface, if only just to sign things (itâs not all digital yet), or to eat a snack. So my solution is to have pullouts. If one needs them, pull them out. But one canât leave them pulled out, so nothing can accumulate on them:
These days I donât deal with paper much. But whenever something does come across my desk, I like to file it. So behind my desk I have an array of drawers—with the little hack that thereâs a slot at the top of each drawer that allows me to immediately slide things into the drawer, without opening it:
I used to fill up a bankerâs box with filed papers every couple of months; now it seems to take a couple of years. And perhaps as a sign of how paperless Iâve become, I have a printer under my desk that I use so rarely that I now seem to go through a ream of paper only every year or so.
There are also other things that have changed over the years. I always want my main computer to be as powerful as possible. And for years that meant that it had to have a big fan to dissipate heat. But since I really like my office to be perfectly quiet (it adds a certain calmness that helps my concentration), I had to put the CPU part of my computer in a different room. And to achieve this, I had a conduit in the floor, through which I had to run often-finicky long-distance video cables. Well, now, finally, I have a powerful computer that doesn’t need a big fan—and so I just keep it behind my desk. (I actually also have three other not-so-quiet computers that I keep in the same room as the treadmill, so that when I’m on the treadmill I can experience all three main modern computing environments, choosing between them with a KVM switch.)
When I mention to people that I’m a remote CEO, they often say, “You must do lots of videoconferencing”. Well, actually, I do basically no videoconferencing. Screensharing is great, and critical. But typically I find video distracting. Often I’ll do a meeting where I have lots of people in case we need to get their input. But for most of the meeting I don’t need all of them to be paying attention (and I’m happy if they’re getting other work done). But if video is on, seeing people who are not paying attention just seems to viscerally kill the mood of almost any meeting.
Given that I don’t have video, audio is very important, and I’m quite a stickler for audio quality in meetings. No speakerphones. No bad cellphone connections. I myself remain quite old school. I wear a headset (with padding added to compensate for my lack of top-of-head hair) with a standard boom microphone. And—partly out of caution about having a radio transmitter next to my head all day—my headset is wired, albeit with a long wire that lets me roam around my office.
Even though I don’t use “talking head” video for meetings, I do have a document camera next to my computer. One time I’ll use this is when we’re talking about phones or tablets. Yes, I could connect their video directly into my computer. But if we’re discussing user experience on a phone it’s often helpful to be able to actually see my finger physically touching the phone.
The document camera also comes in handy when I want to show pages from a physical book, or artifacts of various kinds. When I want to draw something simple I’ll use the annotation capabilities of our screensharing system. But when I’m trying to draw something more elaborate I’ll usually do the retro thing of putting a piece of paper under the document camera, then just using a pen. I like the fact that the image from the document camera comes up in a window on my screen, that I can resize however I want. (I periodically try using drawing tablets but I don’t like the way they treat my whole screen as a canvas, rather than operating in a window that I can move around.)
On the Move
In some ways I lead a simple life, mostly at my desk. But there are plenty of times when Iâm away from my desk—like when Iâm someplace else in my house, or walking outside. And in those cases Iâll normally take a 13″ laptop to use. When I go further afield, it gets a bit more complicated.
If I’m going to do serious work, or give a talk, I’ll take the 13″ laptop. But I never like to be computerless, and the 13″ laptop is a heavy thing to lug around. So instead I also have a tiny 2-lb laptop, which I put in a little bag (needless to say, both the bag and the computer are adorned with our Spikey logo):
And for at least the past couple of years—unless Iâm bringing the bigger computer, usually in a backpack—I have taken to “wearing” my little computer wherever I go. I originally wanted a bag where the computer would fit completely inside, but the nicest bag I could find had the computer sticking out a bit. To my surprise, though, this has worked well. And itâs certainly amusing when Iâm talking to someone and quickly “draw” my computer, and they look confused, and ask, “Where did that come from?”
I always have my phone in my pocket, and if I have just a few moments that’s what I’ll pull out. It works fine if I’m checking mail, and deleting or forwarding a few messages. If I actually want to write anything serious, though, out will come my little computer, with its full keyboard. Of course, if I’m standing up it’s pretty impractical to try to balance the computer on one hand and type with the other. And sometimes if I know I’m going to be standing for a while, I’ll bring a tablet with me. But other times, I’ll just be stuck with my phone. And if I run out of current things I can usefully do (or I don’t have an internet connection) I’ll typically start looking at the “things to read” folder that I maintain synched on all my devices.
Back in 2007 I invented WolframTones because I wanted to have a unique ringtone for my phone. But while WolframTones has been successful as an example of algorithmic music composition, the only trace of it on my phone is the image of WolframTones compositions that I use as my home screen:
How do I take notes when I’m “out and about”? I’ve tried various technological solutions, but in the end none have proved both practical and universally socially acceptable. So I’ve kept doing the same thing for 40 years: in my pocket I have a pen, together with a piece of paper folded three times (so it’s about the size of a credit card). It’s very low-tech, but it works. And when I come back from being out I always take a few moments to transcribe what I wrote down, send out emails, or whatever.
I have little “tech survival kits” that I bring with me. Here are the current contents from my backpack:
The centerpiece is a tiny charger, that charges both my computer (through USB-C) and my phone. I bring various connectors, notably so I can connect to things like projectors. I also bring a very light 2- to 3-prong power adaptor, so I donât find my charger falling out of overused power outlets.
When I’m going on “more serious expeditions” I’ll add some things to the kit:
Thereâs a “charging brick” (unfortunately now in short supply) thatâll keep my computer going for many hours. For events like trade shows, Iâll bring a tiny camera that takes pictures every 30 seconds, so I can remember what I saw. And if Iâm really going out into the wilds, Iâll bring a satphone as well. (Of course, I always have other stuff too, like a very thin and floppy hat, a light neoprene bag-within-a-bag, glasses wipes, hand sanitizer, mosquito wipes, business cards, pieces of chocolate, etc.)
In my efforts to keep organized on trips, I’ll typically pack several plastic envelopes:
In “Presentation” thereâll be the adaptors (VGA, HDMI, …) I need to connect to projectors. Sometimes thereâll be a wired Ethernet adaptor. (For very low-key presentations, Iâll also sometimes bring a tiny projector too.) In “Car” thereâll be a second cellphone that can be used as a GPS, with a magnetic back and a tiny thing for attaching to the air vent in a car. Thereâll be a monaural headset, a phone charger, and sometimes a tiny inverter for my computer. If Iâm bringing the satphone, thereâll also be a car kit for it, with an antenna that magnets to the roof of the car, so it can “see” the satellites. In “Hotel” thereâll be a binaural headset, a second computer charger, and a disk with an encrypted backup of my computer, in case I lose my computer and have to buy and configure a new machine. The fourth plastic envelope is used to store things I get on the trip, and it contains little envelopes—approximately one for each day of my trip—in which I put business cards.
Years ago, I always used to bring a little white-noise fan with me, to mask background noise, particularly at night. But at some point I realized that I didn’t need a physical fan, and instead I just have an app that simulates it (I used to use pink noise, but now I just use “air conditioner sound”). It’s often something of a challenge to predict just how loud the outside noise one’s going to encounter (say, the next morning) will be, and so how loud one should set the masking sound. And, actually, as I write this, I realize I should use modern audio processing in the Wolfram Language to just listen to external sounds, and adjust the masking sound to cover them.
Another thing I need when I travel is a clock. And nowadays it’s just a piece of Wolfram Language code running on my computer. But because it’s software, it can have a few extra features. I always leave my computer on my home timezone, so the “clock” has a slider to specify local time (yes, if I’m ever in a half-hour timezone again I’ll have to tweak the code). It also has a button Start sleep timer. When I press it, it starts a count-up timer, which lets me see how long I’ve been asleep, whatever my biological clock may say. (Start sleep timer also sends an email which gives my assistant an idea of whether or not I’ll make it to that early-next-morning meeting. The top right-hand “mouse corner” is a hack for preventing the computer from going to sleep.)
Whenever itâs practical, I like to drive myself places. It was a different story before cellphones. But nowadays if Iâm driving Iâm productively making a phone call. Iâll have meetings that donât require me to look at anything scheduled for my “drive times” (and, yes, itâs nice to have standard conference call numbers programmed in my phone, so I can voice-dial them). And I maintain a “call-while-driving” list of calls that I can do while driving, particularly if Iâm in an unusual-for-me timezone.
I’ve always had the problem that if I try to work on a computer while I’m being driven by someone else, I get car sick. I thought I had tried everything. Big cars. Little cars. Hard suspension. Soft suspension. Front seat. Back seat. Nothing worked. But a couple of years ago, quite by chance, I tried listening to music with big noise-canceling headphones—and I didn’t get car sick. But what if when I’m being driven I want to be on the phone while I’m using my computer? Well, at the 2018 Consumer Electronics Show, despite my son’s admonition that “just because you can’t tell what they’re selling at a booth doesn’t mean it’s interesting”, I stopped at a booth and got these strange objects, which, despite looking a bit odd, do seem to prevent car sickness for me, at least much of the time:
Giving Talks
I give quite a lot of talks—to a very wide range of audiences. I particularly like giving talks about subjects I havenât talked about before. I give talks to the fanciest business, tech and science groups. I give talks to schoolkids. I enjoy interacting with audiences (Q&A is always my favorite part), and I enjoy being spontaneous. And I essentially always end up doing livecoding.
When I was young I traveled quite a bit. I did have portable computers even back in the 1980s (my first was an Osborne 1 in 1981), though mostly in those days my only way to stay computer-productive was to have workstation computers shipped to my destinations. Then in the early 1990s, I decided I wasn’t going to travel anymore (not least because I was working so intensely on A New Kind of Science). So for a while I basically didn’t give any talks. But then technology advanced. And it started being realistic to give talks through videoconferencing.
I went through several generations of technology, but a number of years ago I built out a videoconferencing setup in my basement. The “set” can be reconfigured in various ways (podium, desk, etc.) But basically I have a back-projection screen on which I can see the remote audience. The camera is in front of the screen, positioned so I’m looking straight at it. If I’m using notes or a script (which, realistically, is rare) I have a homemade teleprompter consisting of a half-silvered mirror and a laptop that I can look at the camera through.
While it’s technically feasible for me to be looking straight at the camera when I’m livecoding, this makes it look to the audience as if I’m staring off into space, which seems weird. It’s better to look slightly down when I’m obviously looking at a screen. And in fact with some setups it’s good for the audience to see the top of a computer right at the bottom of the screen, to “explain” what I’m looking at.
Videoconferenced talks work quite well in many settings (and, for some extra fun, I’ve sometimes used a telepresence robot). But in recent years (partly as a result of my children wanting to do it with me) I’ve decided that traveling is OK—and I’ve been all over the place:
I’ll usually be giving talks—often several per day. And I’ve gradually developed an elaborate checklist of what’s needed to have them work. A podium that’s at the right height and horizontal enough to let me type easily on my computer (and preferably not so massive that I’m hidden from the audience). An attachable microphone that leaves my hands free to type. A network connection that lets me reach our servers. And, of course, to let the audience actually see things, a computer projector.
I remember the very first computer projector I used, in 1980. It was a Hughes “liquid crystal light valve”, and once I got it connected to a CRT terminal, it worked beautifully. In the years since then I’ve used computer projectors all over the world, both in the fanciest audiovisual situations, and in outlying places with ancient equipment and poor infrastructure. And it’s amazing how random it is. In places where one can’t imagine the projector is going to work, it’ll be just fine. And in places where one can’t imagine it won’t work, it’ll fail horribly.
Some years ago I was giving a talk at TED—with some of the fanciest audiovisual equipment I’d ever seen. And that was one of the places where things failed horribly. Fortunately we did a test the day before. But it took a solid three hours to get the top-of-the-line computer projector to successfully project my computer’s screen.
And as a result of that very experience I decided I’d better actually understand how computers talk to projectors. It’s a complicated business, that involves having the computer and the projector negotiate to find a resolution, aspect ratio, frame rate, etc. that will work for both of them. Underneath, there are things called EDID strings that are exchanged, and these are what typically get tangled up. Computer operating systems have gotten much better about handling this in recent years, but for high-profile, high-production-value events, I have a little box that spoofs EDID strings to force my computer to send a specific signal, regardless of what the projector seems to be asking it for.
Some of the talks I give are completely spontaneous. But often I’ll have notes—and occasionally even a script. And I’ll always write these in a Wolfram Notebook. I then have code that “paginates” them, basically replicating “paragraphs” at the end of each page, so I have freedom in when I “turn the page”. In past years I used to transfer these notes to an iPad that I’d set up to “turn the page” whenever I touched its screen. But in recent years I’ve actually just synched files, and used my little computer for my notes—which has the advantage that I can edit them right up to the moment I start giving the talk.
In addition to notes, I’ll sometimes also have material that I want to immediately bring into the talk. Now that we have our new Presenter Tools system, I may start creating more slide-show-like material. But that’s not how I’ve traditionally worked. Instead, I’ll typically just have a specific piece of Wolfram Language code I want to input, without having to take the time to explicitly type it. Or perhaps I’ll want to pick an image from a “slide farm” that I want to immediately put on the screen, say in response to a question. (There’s a lot of trickiness about projector resolutions in, for example, slides of cellular automata, because unless they’re “pixel perfect” they’ll alias—and it’s not good enough just to scale them like typical slide software would.)
So how do I deal with bringing in this material? Well, I have a second display connected to my computer—whose image isn’t projected. (And, yes, this can contribute to horrible tangling of EDID strings.) Then on that second display I can have things to click or copy. (I have a Wolfram Language function that will take a notebook of inputs and URLs, and make me a palette that I can click to type inputs, open webpages, etc.)
In the past we used to have a little second monitor to attach to my laptop—essentially a disembodied laptop screen. But it took all sorts of kludges to get both it and the projector connected to my laptop (sometimes one would be USB, one would be HDMI, etc.) But now we can just use an iPad—and it’s all pure software (though the interaction with projectors can still be finicky):
For a while, just to be stylish, I was using a computer with a Spikey carved out of its case, and backlit. But the little rhombuses in it were a bit fragile, so nowadays I mostly just use “Spikey skins” on my computers:
My Filesystem
The three main applications I use all day are Wolfram Desktop, a web browser, and email. My main way of working is to create (or edit) Wolfram Notebooks. Here are a few notebooks I worked on today:
On a good day I’ll type at least 25,000 characters into Wolfram Notebooks (and, yes, I record all my keystrokes). I always organize my notebooks into sections and subsections and so on (which, very conveniently, automatically exist in hierarchical cells). Sometimes I’ll write mostly text in a notebook. Sometimes I’ll screen capture something from elsewhere and paste it in, as a way to keep notes. Depending on what I’m doing, I’ll also actually do computations in a notebook, entering Wolfram Language input, getting results, etc.
Over the years, I’ve accumulated over a hundred thousand notebooks, representing product designs, plans, research, writings, and, basically, everything I do. All these notebooks are ultimately stored in my filesystem (yes, I sync with the cloud, use cloud files, and file servers, etc.) And I take pains to keep my filesystem organized—with the result I can typically find any notebook I’m looking for just by navigating my filesystem, faster than I could formulate a search for it.
I believe I first thought seriously about how to organize my files back in 1978 (which was also when I started using the Unix operating system). And over the past 40 years I’ve basically gone through five generations of filesystem organization, with each generation basically being a reflection of how I’m organizing my work at that stage in my life.
For example, during the period from 1991 to 2002 when I was writing my big book A New Kind of Science, a substantial part of my filesystem was organized simply according to sections of the book:
And itâs very satisfying that today I can go immediately from, say, an image in the online version of the book, to the notebook that created it (and the stability of the Wolfram Language means that I can immediately run the code in the notebook again—though sometimes it can now be written in a more streamlined way).
The sections of the book are basically laid out in the NewScience/Book/Layout/ folder of my “third-generation” filesystem. Another part of that filesystem is NewScience/BookResearch/Topics. And in this folder are about 60 subfolders named for broad topics that I studied while working on the book. Within each of these folders are then further subfolders for particular projects I did while studying those topics—which often then turned into particular sections or notes in the book.
Some of my thinking about computer filesystems derives from my experience in the 1970s and 1980s with physical filesystems. Back when I was a teenager doing physics I voraciously made photocopies of papers. And at first I thought the best way to file these papers would be in lots of different categories, with each category stored in a different physical file folder. I thought hard about the categories, often feeling quite pleased with the cleverness of associating a particular paper with a particular category. And I had the principle that if too many papers accumulated in one category, I should break it up into new categories.
All this at first seemed like a good idea. But fairly quickly I realized it wasn’t. Because too often when I wanted to find a particular paper I couldn’t figure out just what cleverness had caused me to associate it with what category. And the result was that I completely changed my approach. Instead of insisting on narrow categories, I allowed broad, general categories—with the result that I could easily have 50 or more papers filed in a single category (often ending up with multiple well-stuffed physical file folders for a given category):
And, yes, that meant that I would sometimes have to leaf through 50 papers or more to find one I wanted. But realistically this wouldnât take more than a few minutes. And even if it happened several times a day it was still a huge win, because it meant that I could actually successfully find the things I wanted.
I have pretty much the same principle about some parts of my computer filesystem today. For example, when I’m collecting research about some topic, I’ll just toss all of it into a folder named for that topic. Sometimes I’ll even do this for years. Then when I’m ready to work on that topic, I’ll go through the folder and pick out what I want.
These days my filesystem is broken into an active part (that I continuously sync onto all my computers), and a more archival part, that I keep on a central fileserver (and that, for example, contains my older-generation filesystems).
There are only a few top-level folders in my active filesystem. One is called Events. Its subfolders are years. And within each year I’ll have a folder for each of the outside events I go to in that year. In that folder I’ll store material about the event, notebooks I used for talks there, notes I made at the event, etc. Since in a given year I won’t go to more than, maybe, 50 events, it’s easy to scan through the Events folder for a given year, and find the folder for a particular event.
Another top-level folder is called Designs. It contains all my notes about my design work on the Wolfram Language and other things we’re building. Right now there are about 150 folders about different active areas of design. But there’s also a folder called ARCHIVES, which contains folders about earlier areas that are no longer active.
And in fact this is a general principle in the project-oriented parts of my filesystem. Every folder has a subfolder called ARCHIVES. I try to make sure that the files (or subfolders) in the main folder are always somehow active or pending; anything that’s finished with I put in ARCHIVES. (I put the name in capitals so it stands out in directory listings.)
For most projects I’ll never look at anything in ARCHIVES again. But of course it’s easy to do so if I want to. And the fact that it’s easy is important, because it means I don’t have nagging concerns about saying “this is finished with; let’s put it in ARCHIVES”, even if I think there’s some chance it might become active again.
As it happens, this approach is somewhat inspired by something I saw done with physical documents. When I was consulting at Bell Labs in the early 1980s I saw that a friend of mine had two garbage cans in his office. When I asked him why, he explained that one was for genuine garbage and the other was a buffer into which he would throw documents that he thought he’d probably never want again. He’d let the buffer garbage can fill up, and once it was full, he’d throw away the lower documents in it, since from the fact that he hadn’t fished them out, he figured he’d probably never miss them if they were thrown away permanently.
Needless to say, I don’t follow exactly this approach, and in fact I keep everything, digital or paper. But the point is that the ARCHIVES mechanism gives me a way to easily keep material while still making it easy to see everything that’s active.
I have a bunch of other conventions too. When I’m doing designs, I’ll typically keep my notes in files with names like Notes-01.nb or SWNotes-01.nb. It’s like my principle of not having too many file categories: I don’t tend to try to categorize different parts of the design. I just sequentially number my files, because typically it’ll be the most recent—or most recent few—that are the most relevant when I continue with a particular design. And if the files are just numbered sequentially, it’s easy to find them; one’s not trying to remember what name one happened to give to some particular direction or idea.
A long time ago I started always naming my sequential files file-01, file-02, etc. That way pretty much any sorting scheme will sort the files in sequence. And, yes, I do often get to file-10, etc. But in all these years I have yet to get even close to file-99.
Knowing Where to Put Everything
When Iâm specifically working on a particular project, Iâll usually just be using files in the folder associated with that project. But on a good day, Iâll have lots of ideas about lots of different projects. And I also get hundreds of emails every day, relevant to all sorts of different projects. But often itâll be months or years before Iâm finally ready to seriously concentrate on one of these other projects. So what I want to do is to store the material I accumulate in such a way that even long in the future I can readily find it.
For me, there are typically two dimensions to where something should be stored. The first is (not surprisingly) the content of what it’s about. But the second is the type of project in which I might use it. Is it going to be relevant to some feature of some product? Is it going to be raw material for some piece I write? Is it a seed for a student project, say at our annual Summer School? And so on.
For some types of projects, the material I’m storing typically consists of a whole file, or several files. For others, I just need to store an idea which can be summarized in a few words or paragraphs. So, for example, the seed for a student project is typically just an idea, that I can describe with a title, and perhaps a few lines of explanation. And in any given year I just keep adding such project ideas to a single notebook—which, for example, I’ll look at—and summarize—right before our annual summer programs.
For pieces like this that I’m potentially going to write, it’s a little different. At any given time, there are perhaps 50 pieces that I’m considering at some point writing. And what I do is to create a folder for each of them. Each will typically have files with names like Notes-01.nb, into which I accumulate specific ideas. But then the folder will also contain complete files, or groups of files, that I accumulate about the topic of the piece. (Sometimes I’ll organize these into subfolders, with names like Explorations and Materials.)
In my filesystem, I have folders for different types of projects: Writings, Designs, StudentProjects, etc. I find it important to have only a modest number of such folders (even with my fairly complex life, not much more than a dozen). When something comes in—say from a piece of email, or from a conversation, or from something I see on the web, or just from an idea I have—I need to be able to quickly figure out what type of project (if any) it might be relevant to.
At some level it’s as simple as “what file should I put it into”? But the key point is to have a pre-existing structure that makes it quick to decide that—and then to have this structure be one in which I can readily find things even far into the future.
There are plenty of tricky issues. Particularly if years go by, the way one names or thinks about a topic may change. And sometimes that means at some point I’ll just rename a folder or some such. But the crucial thing as far as I’m concerned is that at any given time the total number of folders into which I’m actively putting things is small enough that I can basically remember all of them. I might have a dozen folders for different types of projects. Then some of these will need subfolders for specific projects about specific topics. But I try to limit the total number of “active accumulation folders” to at most a few hundred.
Some of those “accumulation folders” I’ve had for a decade or more. A few will come into existence and be gone within a few months. But most will last at most a few years—basically the time between when I conceptualize a project, and when the project is, for practical purposes, finished.
It’s not perfect, but I end up maintaining two hierarchies of folders. The first, and most important, is in my filesystem. But the second is in my email. There are two basic reasons I maintain material in email folders. The first is immediate convenience. Some piece of mail comes in and I think “that’s relevant to such-and-such a project that I’m planning to do”—and I want to store it in an appropriate place. Well, if that place is a mail folder, all I have to do is move the mail with one mouse motion (or maybe with one press of a Touch Bar button). I don’t have to, for example, find a file or filesystem folder to put it into.
There’s also another reason it’s good to leave mail as mail: threading. In the Wolfram Language we’ve now got capabilities both for importing mailboxes, and for connecting to live mail servers. And one of the things one quickly sees is how complicated the graphs (actually, hypergraphs) of email conversations can be. Mail clients certainly aren’t perfect as a way to view these conversations, but it’s a lot better to use one than, say, to have a collection of separate files.
When projects are fairly well defined, but aren’t yet very active, I tend to use filesystem folders rather than email folders. Typically what will be coming in about these projects are fairly isolated (and non-threaded) pieces of mail. And I find it best either just to drag those pieces of mail into appropriate project folders, or to copy out their contents and add them to notebooks.
When a project is very active, there may be lots of mail coming in about it, and it’s important to preserve the threading structure. And when a project isn’t yet so well defined, I just want to throw everything about it into a single “bucket”, and not have to think about organizing it into subfolders, notebooks, etc.
If I look at my mail folders, I see many that parallel folders in my filesystem. But I see some that do not, particularly related to longer-term project concepts. And I have many such folders that have been there for well over a decade (my current overall mail folder organization is about 15 years old). Sometimes their names aren’t perfect. But there are few enough folders, and I’ve seen them for long enough, that I have a sense of what I’m filing in them, even though their names don’t quite capture it.
It’s always very satisfying when I’m ready to work on a project, and I open the mail folder for it, and start going through messages, often from long ago. Just in the past few weeks, as we wrap up a major new version of the Wolfram Language, I’m starting to look ahead, and I’ve been going through folders with messages from 2005, and so on. When I saved those messages, I didn’t yet have a definite framework for the project they’re about. But now I do. So when I go through the messages I can quickly put them into the appropriate active notebooks and so on. Then I delete the messages from the mail folder, and eventually, once it is empty, delete the whole mail folder. (Unlike with files, I don’t find it useful to have an ARCHIVES folder for mail; the mail is just too voluminous and not organized enough, so to find any particular item I’ll probably end up having to search for it anyway, and of course I certainly have all of my mail stored.)
OK, so I have my filesystem, and I have mail. At our company we also have an extensive project management system, as well as all sorts of databases, request trackers, source control systems, etc. Mostly the nature of my current work does not cause me to interact directly with these, and I don’t explicitly store my own personal output in them. At different times, and with different projects, I have done so. But right now my interaction with these systems is basically only as a viewer, not an author.
Beyond these systems, there are lots of things that I interact with basically through webpages. These might be public sites like wolframalpha.com or wolfram.com. They might be internal sites at our company. And they might be preliminary (say, “test” or “devel”) versions of what will in the future be public websites or web-based services. I have a personal homepage that gives me convenient access to all these things:
The source for the homepage is (needless to say) a Wolfram Notebook. I can edit this notebook in my filesystem, then press a button to deploy a version to the Wolfram Cloud. Iâve got an extension in my web browser so that every time I create a new browser window or tab, the initial content will be my personal homepage.
And when I’m going to start doing something, there are just a few places I go. One is this web homepage, which I access many hundreds of times every day. Another is my email and its folders. Another is my desktop filesystem. And basically the only other one of any significance is my calendar system.
From time to time, I’ll see other people’s computers, and their desktops will be full of files. My desktop is completely empty, and plain white (convenient for whole-screen screensharing and livestreaming). I’d be mortified if there were any files to be seen on my desktop. I’d consider it a sign of defeat in my effort to keep what I’m doing organized. The same can be said of generic folders like Documents and Downloads. Yes, in some situations applications etc. will put files there. But I consider these directories to be throwaways. Nothing in them do I intend to be part of my long-term organizational structure. And they’re not synched to the cloud, or across my different computers.
Whatever the organization of my files may be, one feature of them is that I keep them a long time. In fact, my oldest file dates are from 1980. Back then, there was something a bit like the cloud, except it was called timesharing. I’ve actually lost some of the files that I had on timesharing systems. But the ones I had on on-premise computers are still with me (though, to be fair, some had to be retrieved from 9-track backup tapes).
And today, I make a point of having all my files (and all my email) actively stored on-premise. And, yes, that means I have this in my basement:
The initial storage is on a standard RAID disk array. This is backed up to computers at my company headquarters (about 1000 miles away), where standard tape backups are done. (In all these years, Iâve only ever had to retrieve from a backup tape once.) I also sync my more active files to the cloud, and to all my various computers.
All the Little Conveniences
My major two personal forms of output are mail messages and Wolfram Notebooks. And over the 30 years since we first introduced notebooks weâve optimized our notebook system to the point where I can just press a key to create a default new notebook, and then Iâm immediately off and running writing what automatically becomes a good-looking structured document. (And, by the way, itâs very nice to see that weâve successfully maintained compatibility for 30 years: notebooks I created back in 1988 still just work.)
Sometimes, however, I’m making a notebook that’s not so much for human consumption as for input to some automated process. And for this, I use a whole variety of specially set up notebooks. For example, if I want to create an entry in our new Wolfram Function Repository, I just go to the menu item (available in any Version 12 system) File > New > Repository Item > Function Repository Item:
This effectively “prompts” me for items and sections to add. When Iâm done, I can press Submit to Repository to send the notebook off to our central queue for repository item reviews (and, just because Iâm the CEO doesnât mean I get out of the review process—or want to).
I actually create a fair amount of content that’s structured for further processing. A big category is Wolfram Language documentation. And for authoring this we have an internal system we call DocuTools, that’s all based on a giant palette developed over many years, that I often say reminds one of an airplane cockpit in its complexity:
The idea of DocuTools is to make it as ergonomic as possible to author documentation. It has more than 50 subpalettes (a few shown above), and altogether no less than 1016 buttons. If I want to start a new page for a Wolfram Language function I just press New Function Page, and up pops:
A very important part of this page is the stripe at the top that says “Future”. This means that even though the page will be stored in our source control system, itâs not ready yet: itâs just something weâre considering for the future. And the system that builds our official documentation will ignore the page.
Usually we (which quite often actually means me) will write documentation for a function before the function is implemented. And we’ll include all sorts of details about features the function should have. But when the function is actually first implemented, some of those features may not be ready yet. And to deal with this we (as we call it) âfuturizeâ parts of the documentation, giving it a very visible pink background. It’s still there in the source control system, and we see it every time we look at the source for the documentation page. But it’s not included when the page for documentation that people will see is built.
DocuTools is of course implemented in the Wolfram Language, making extensive use of the symbolic structure of Wolfram Notebooks. And over the years it’s grown to handle many things that aren’t strictly documentation; in fact, for me it’s become the main hub for the creation of almost all notebook-based content.
There’s a button, for example, for Stephen Wolfram Blog. Press it and one gets a standard notebook ready to write into. But in DocuTools there’s a whole array of buttons that allow one to insert suggestions and edits. And when I’ve written a blog what will come back is typically something like this:
The pink boxes are “you really need to fix this”; the tan are “here’s a comment”. Click one and up comes a little form:
Of course, there are plenty of change-tracking and redlining systems out there in the world. But with the Wolfram Language it becomes easy to create a custom one that’s optimized for my needs, so that’s what I’ve had done. Before I had this, it used to take many hours to go through edit suggestions (I remember a horrifying 17-hour plane ride where I spent almost the whole time going through suggestions for a single post). But now—because it’s all optimized for me—I can zip through perhaps 10 times faster.
Very often tools that are custom built for me end up being adapted so everyone else can use them too. An example is a system for authoring courses and creating videos. I wanted to be able to do this as a “one-man band”—a bit like how I do livestreaming. My idea was to create a script that contains both words to say and code to input, then to make the video by screen recording in real time while I went through the script. But how would the inputs work? I couldn’t type them by hand because it would interrupt the real-time flow of what I was saying. But the obvious thing is just to “autotype” them directly into a notebook.
But how should all this be orchestrated? I start from a script:
Then I press Generate Recording Configuration. Immediately a title screen comes up in one area of my screen, and I set up my screen-recording system to record from this area. Elsewhere on my screen is the script. But what about the controls? Well, they’re just another Wolfram Notebook, that happens to act as a palette containing buttons:
But how can I actually operate this palette? I canât use my mouse, because then Iâd take focus away from the notebook thatâs being screen recorded. So the idea that I had is to put the palette on an extended desktop, that happens to be being displayed on an iPad. So then to “perform” the script, I just press buttons on the palette.
There’s a big Advance Script button. And let’s say I’ve read to a point in the script where I need to type something into the notebook. If I want to simulate actual typing I press Slow Type. This will enter the input character-at-a-time into the notebook (yes, we measured the inter-key delay distribution for human typing, and simulate it). After a while it gets annoying to see all that slow typing. So then I just use the Type button, which copies the whole input immediately into the notebook. If I press the button again, it’ll perform its second action: Evaluate. And that’s the equivalent of pressing Shift+Enter in the notebook (with some optional extra explanatory popups suitable for the video).
I could go on about other tools I’ve had built using the Wolfram Language, but this gives a flavor. But what do I use that isn’t Wolfram Language? Well, I use a web browser, and things that can be reached through it. Still, quite often, I’m just going to the Wolfram Cloud, and for example viewing or using cloud notebooks there.
Sometimes I’ll use our public Wolfram Cloud. But more often I’ll use a private Wolfram Cloud. The agendas for most of our internal meetings are notebooks that are hosted on our internal Wolfram Cloud. I also personally have a local private Wolfram Cloud running, that I host an increasing number of applications on.
Here’s the dock on my computer as of right now:
It’s got a filesystem browser; it’s got an email client; it’s got three web browsers (yes, I like to test our stuff on multiple browsers). Then I’ve got a calendar client. Next is the client for our VoIP phone system (right now I’m alternating between using this, and using audio along with our screensharing system). Then, yes, at least right now I have a music app. I have to say it’s rather rare that my day gives me a chance to listen to music. Probably the main time when I end up doing it is when I’m very behind on email, and need something to cheer me up as I grind through thousands of messages. As soon as I’m actually writing anything nontrivial, though, I have to pause the music, or I can’t concentrate. (And I have to find music without vocals—because I’ve noticed I can’t read at full speed if I’m hearing vocals.)
Sometimes I’ll end up launching a standard word processor, spreadsheet, etc. app because I’m opening a document associated with one of these apps. But I have to admit that in all these years I’ve essentially never authored a document from scratch with any of these apps; I end up just using technology of ours instead.
Occasionally I’ll open a terminal window, and directly use operating system commands. But this is becoming less and less common—because more and more I’m just using the Wolfram Language as my “super shell”. (And, yes, it’s incredibly convenient to store and edit commands in a notebook, and to instantly be able to produce graphical and structured output.)
As I write this, I realize a little optimization I haven’t yet made. On my personal homepage there are some links that do fairly complex things. One, for example, initiates the process for me doing an unscheduled livestream: it messages our 24/7 system monitoring team so they can take my feed, broadcast it, and monitor responses. But I realize that I still have quite a few custom operating system commands, that do things like update from the source code repository, that I type into a terminal window. I need to set these up in my private cloud, so I can just have links on my personal homepage that run Wolfram Language code for these commands. (To be fair, some of these commands are very old; for example, my fmail command that sends a mail message in the future, was written nearly 30 years ago.)
But, OK, if I look at my dock of apps, there’s a definite preponderance of Spikey ones. But why, for example, do I need three identical standard Spikeys? They’re all the Wolfram Desktop app. But there are three versions of it. The first one is our latest distributed version. The second one is our latest internal version, normally updated every day. And the third one (which is in white) is our “prototype build”, also updated every day, but with lots of “bleeding edge” features that aren’t ready to go into serious testing.
It requires surprisingly fancy operating system footwork to get these different versions installed every night, and to correctly register document types with them. But it’s very important to my personal workflow. Typically I’ll use the latest internal version (and, yes, I have a directory with many previous versions too), but occasionally, say for some particular meeting, I’ll try out the prototype build, or I’ll revert to the released build, because things are broken. (Dealing with multiple versions is one of those things that’s easier in the cloud—and we have a whole array of different configurations running in internal private clouds, with all sorts of combinations of kernel, front end, and other versions.)
When I give talks and so on, I almost always use the latest internal version. I find that livecoding in front of an audience is a great way to find bugs—even if it sometimes makes me have to explain, as I put it, the “disease of the software company CEO”: to always want to be running the latest version, even if it hasn’t been seriously tested and was built the night before.
Archiving & Searching
A critical part of my personal infrastructure is something that in effect dramatically extends my personal memory: my “metasearcher”. At the top of my personal homepage is a search box. Type in something like “rhinoceros elephant” and Iâll immediately find every email Iâve sent or received in the past 30 years in which thatâs appeared, as well as every file on my machine, and every paper document in my archives:
To me itâs extremely convenient to have a count of the messages by year; it often helps me remember the history or story behind whatever Iâm asking. (In this case, I can see a peak in 2008, which is when we were getting ready to launch Wolfram|Alpha—and I was working on data about lots of kinds of things, including species.)
Of course, a critical piece of making my metasearcher work is that I’ve stored so much stuff. For example, I actually have all the 815,000 or so emails that I’ve written in the past 30 years, and all the 2.3 million (mostly non-spam) ones I’ve received. And, yes, it helps tremendously that I’ve had a company with organized IT infrastructure etc. for the past 32 years.
But email, of course, has the nice feature that it’s “born digital”. What about things that were, for example, originally on paper? Well, I have been something of an “informational packrat” for most of my life. And in fact I’ve been pretty consistently keeping documents back to when I started elementary school in 1968. They’ve been re-boxed three times since then, and now the main ones are stored like this:
(I also have file folder storage for documents on people, organizations, events, projects and topics.) My rate of producing paper documents increased through about 1984, then decayed quite rapidly, as I went more digital. Altogether I have about a quarter million pages of primary non-bulk-printed documents—mostly from the earlier parts of my life.
About 15 years ago I decided I needed to make these searchable, so I initiated the project of scanning all of them. Most of the documents are one or a few pages in length, so they can’t be handled by an automatic feeder—and so we set up a rig with a high-resolution camera (and in those days it needed flash). It took several person-years of work, but eventually all the documents were scanned.
We automatically cropped and white-balanced them (using Wolfram Language image processing), then OCR’ed them, and put the OCR’ed text as a transparent layer into the scanned image. If I now search for “rhinoceros” I find 8 documents in my archive. Perhaps not surprisingly given that search term, they’re a bit random, including for example the issue of my elementary school magazine from Easter 1971.
OCR works on printed text. But what about handwritten text? Correspondence, even if it’s handwritten, usually at least comes on printed letterhead. But I have many pages of handwritten notes with basically nothing printed on them. Recognizing handwriting purely from images (without the time series of strokes) is still beyond current technology, but I’m hoping that our neural-net-based machine learning systems will soon be able to tackle it. (Conveniently, I’ve got quite a few documents where I have both my handwritten draft, and a typed version, so I’m hoping to have a training set for at least my personal handwriting.)
But even though I can’t search for handwritten material, I can often find it just by “looking in the right box”. My primary scanned documents are organized into 140 or so boxes, each covering a major period or project in my life. And for each box, I can pull up thumbnails of pages, grouped into documents. So, for example, here are school geography notes from when I was 11 years old, together with the text of a speech I gave:
I have to say that pretty much whenever I start looking through my scanned documents from decades ago I end up finding something unexpected and interesting, that very often teaches me something about myself, and about how I ended up developing in some particular direction.
It may be something fairly specific to my life, and the fact that I’ve worked on building long-term things, as well as that I’ve kept in touch with a large number of people over a long period of time, but I’m amazed by the amount of even quite ancient personal history that I seem to encounter practically every day. Some person or some organization will contact me, and I’ll look back at information about interactions I had with them 35 years ago. Or I’ll be thinking about something, and I’ll vaguely remember that I worked on something similar 25 years ago, and look back at what I did. I happen to have a pretty good memory, but when I actually look at material from the past I’m always amazed at how many details I’ve personally forgotten.
I first got my metasearcher set up nearly 30 years ago. The current version is based on Wolfram Language CreateSearchIndex/TextSearch functionality, running on my personal private cloud. It’s using UpdateSearchIndex to update every few minutes. The metasearcher also “federates in” results from APIs for searching our corporate websites and databases.
But not everything I want can readily be found by search. And another mechanism I have for finding things is my “personal timeline”. I’ve been meaning for ages to extend this, but right now it basically just contains information on my external events, about 40 of them per year. And the most important part is typically my “personal trip report”, which I meticulously write, if at all possible within 24 hours.
Usually the trip report is just text (or at least, text structured in a notebook). But when I go to events like trade shows I typically bring a tiny camera with me, that takes a picture every half-minute. If I’m wearing one of those lanyard name tags I’ll typically clip the camera on the top of the name tag, among other things putting it at an ideal height to capture name tags of people I meet. When I write my personal trip report I’ll typically review the pictures, and sometimes copy a few into my trip notebook.
But even with all my various current sources of archival material (which now include chat messages, livestreams, etc.), email still remains the most important. Years ago I decided to make it easy for people to find an email address for me. My calculation was that if someone wants to reach me, then in modern times they’ll eventually find a way to do it, but if it’s easy for them just to send email, that’s how they’ll contact me. And, yes, having my email address out there means I get lots of email from people I don’t know around the world. Some of it is admittedly strange, but a lot is interesting. I try to look at all of it, but it’s also sent to a request tracker system, so my staff can make sure important things get handled. (It is sometimes a little odd for people to see request tracker ticket metadata like SWCOR #669140 in email subject lines, but I figure it’s a small price to pay for making sure the email is actually responded to.)
I might mention that for decades email has been the primary means of communication inside our (geographically distributed) company. Yes, we have project management, source control, CRM and other systems, as well as chat. But at least for the parts of the company that I interact with, email is overwhelmingly dominant. Sometimes it’s individual emails being sent between people. Sometimes it’s email groups.
It’s been a running joke for a long time that we have more email groups than employees. But we’ve been careful to organize the groups, for example identifying different types by prefixes to their names (t- is a mailing list for a project team, d- a mailing list for a department, l- a more open mailing list, r- a mailing list for automated reports, q- a request list, etc.) And for me at least this makes it plausible to remember what the right list is for some mail I want to send out.
Databases of People & Things
I know a lot of people, from many different parts of my life. Back in the 1980s I used to just keep a list of them in a text file (before then it was a handwritten address book). But by the 1990s I decided I needed to have a more systematic database for myself—and created what I started calling pBase. In recent years the original technology of pBase began to seem quite paleolithic, but I now have a modern implementation using the Wolfram Language running in my personal private cloud.
Itâs all quite nice. I can search for people by name or attributes, or—if Iâm for example going to be visiting somewhere—I can just have pBase show me a map of our latest information about whoâs nearby:
How does pBase relate to social networks? I’ve had a Facebook account for a long time, but it’s poorly curated, and always seems to ride at the maximum number of possible friends. LinkedIn I take much more seriously, and make a point of adding people only if I’ve actually talked to them (I currently have 3005 connections, so, yes, I’ve talked to quite a few people).
It’s very convenient that every so often I can download data from my LinkedIn account via ServiceExecute to update what’s in pBase. But LinkedIn captures only a fraction of people I know. It doesn’t include many of my more prominent friends and acquaintances, as well as most academics, many students, etc.
Eventually I’ll probably get pBase developed more, and perhaps make the technology generally available. But within our company, there’s already a system that illustrates some potential aspirations: our internal company directory—which is running in our internal private cloud, and basically uses Wolfram|Alpha-style natural language understanding to let one ask natural language questions.
I might mention in addition to our company directory, we also maintain another database that I, at least, find very useful, particularly when I’m trying to figure out who might know the answer to some unusual question, or who we might tap for some new project. We call it our Who Knows What database. And for each person it gives a profile of experience and interests. Here’s the entry for me (and here’s the source with the question details):
In terms of personal databases, another useful one for me is the database of books I own. I havenât been buying too many books in the past decade or so, but before then I accumulated a library of about 6000 volumes, and itâs not uncommon—particularly when Iâm doing more historically oriented research—that Iâll want to consult quite a few of them. But how should they be organized? “Big” classification schemes like Dewey Decimal or Library of Congress are overkill, and donât do a great job of matching my personal “cognitive map” of topics.
Like my filesystem folders, or my physical folders of papers, I’ve found the best scheme is to put the books into fairly broad categories—small enough in number that I can spatially remember where they are in my library. But how should books be arranged within a category?
Well, here I get to tell a cautionary tale (that my wife regularly uses as an example) of what can go wrong in my kind of approach. Always liking to understand the historical progression of ideas, I thought it would be nice to be able to browse a category of books on a shelf in historical order (say, by first publication date). But this makes it difficult to find a specific book, or, for example, to reshelve it. (It would be easier if books had their publication dates printed on their spines. But they don’t.)
About 20 years ago I was preparing to move all my books to a new location, with different lengths of shelves. And I had the issue of trying to map out how to arrange book categories on the new shelves (“how many linear feet is quantum field theory and where can it fit in?”) So I thought: “Why not just measure the width of each book, and while I’m at it also measure its height and its color?” Because my idea was that then I could make a graphic of each shelf, with books shown with realistic widths and colors, then put an arrow in the graphic to indicate the location (easily identified visually from “landmarks” of other books) of a particular book.
I got a colorimeter (it was before ubiquitous digital cameras) and started having the measurements made. But it turned out to be vastly more labor-intensive than expected, and, needless to say, didn’t get finished before the books had to be moved. Meanwhile, the day the books were moved, it was noticed that the packing boxes fit more books if one didn’t just take a single slab of books off a shelf, but instead put other books around the edges.
The result was that 5100 books arrived, basically scrambled into random order. It took three days to sort them. And at this point, I decided just to keep things simpler, and alphabetize by author in each category. And this certainly works fine in finding books. But one result of my big book inventory project is that I do now have a nice, computable version of at least all the books connected to writing A New Kind of Science, and it’s actually in the Wolfram Data Repository:
|
✕
ResourceData["Books in Stephen Wolfram's Library"] |
Personal Analytics
In 2012 I wrote a piece about personal analytics and the data Iâve collected on myself. Back then I had about a third of a million outgoing emails in my archive; now itâs half a million more, and I can extend my diurnal plot of email I’ve sent:
(The big empty spaces are when Iâm asleep, and, yes, as Iâve changed projects—e.g. finishing A New Kind of Science in 2002—my sleep habits have changed; Iâm also now trying an experiment of going to sleep earlier.)
I have systems that keep all sorts of data, including every keystroke I type, every step I take and what my computer screen looks like every minute (sadly, the movie of this is very dull). I also have a whole variety of medical and environmental sensors, as well as data from devices and systems that I interact with.
It’s interesting every so often to pick up those Wolfram Data Drop databins and use them to do some data science on my life. And, yes, in broad terms I find that I am extremely consistent and habitual—yet every day there are different things that happen, that make my “productivity” (as measured in a variety of ways) bounce around, often seemingly randomly.
But one thing about collecting all this data is that I can use it to create dashboards, and these I find useful every single day. For example, running in my private cloud is a monitoring system for my email:
The yellow curve is my total number of pending email messages; the red is the number I havenât even opened yet. These curves are pretty sensitive to all kinds of features of my life, and for example when Iâm intensely working on some project, Iâll often see my email “go to seed” for a little while. But somehow in trying to pace myself and decide when I can do what, I find this email dashboard very helpful.
It’s also helpful that every day I get emails reporting on the previous day. How many keystrokes did I type, and in what applications? What files did I create? How many steps did I take? And so on.
I keep all kinds of health and medical data on myself too, and have done so for a long time. It’s always great to have started measuring something a long time ago, so one can plot a several-decade time series and see if anything’s changed. And, actually, the thing I’ve noticed is that often my value (say blood level) for something has remained numerically essentially the same for years—but many of the “normal ranges” quoted by labs have bounced all over the place. (Realistically this isn’t helped by labs inferring normal ranges from their particular observed populations, etc.)
I got my whole genome sequenced in 2010. And although I haven’t learned anything dramatic from it, it certainly helps me feel connected to genomic research when I can see some SNP variant mentioned in a paper, and I can immediately go look to see if I have it. (With all the various vicissitudes of strands, orientations and build numbers, I tend to stick to first principles, and just look for flanking sequences with StringPosition.)
Like so many of the things I’ve described in this piece, what has worked for me in doing personal analytics is to do what’s easy to do. I’ve never yet quite solved the problem, for example, of recording what I eat (our image identification isn’t yet quite good enough, and even made-just-for-me apps to enter food have always seemed a bit too onerous). But whenever I have a system that just operates automatically, that’s when I successfully collect good personal analytics data. And having dashboards and daily emails helps both in providing ongoing feedback, and in being able to check if something’s gone wrong with the system.
The Path Ahead
I’ve described—in arguably quite nerdy detail—how some of my personal technology infrastructure is set up. It’s always changing, and I’m always trying to update it—and for example I seem to end up with lots of bins of things I’m not using anymore (yes, I get almost every “interesting” new device or gadget that I find out about):
But although things like devices change, I’ve found that the organizational principles for my infrastructure have remained surprisingly constant, just gradually getting more and more polished. And—at least when they’re based on our very stable Wolfram Language system—I’ve found that the same is true for the software systems I’ve had built to implement them.
What of the future? Some things will certainly get upticked. I realized while writing this piece that I can now upgrade to 4k monitors (or higher) without affecting screensharing (the feed is automatically downsampled). Before too long maybe I’ll be using AR to annotate my environment in real time. Maybe eventually I’ll have some way to do XR-based as-if-in-person videoconferencing. Maybe—as I’ve been assuming will be possible for 40+ years—I’ll finally be able to type faster using something like EEG. And so on.
But the more important changes will be in having better-developed, and more automated, workflows. In time I expect it’ll be possible to use our machine learning tools to do automatic “computational history”, for example assembling a useful and appropriately clustered timeline of things I’ve done, say in a particular area.
In my efforts at historical research, I’ve had occasion to use lots of archives of people and organizations. There’s usually a certain amount of indexing and tagging that’s been done. (Who is that letter to and from? When was it written? What are its keywords? Where was it filed? And so on.) But things tend to be very granular, and it’s usually hard work to determine the overall arc of what happened.
My first goal is to make all the material I personally have useful for myself. But I’m thinking of soon starting to open up some of the older material for other people to see. And I’m studying how—in modern times, with all the cloud infrastructure, machine learning, visualization, computational documents, etc. that we have—I can build the best possible system for presenting and exploring archives.
As I think about my day, I ask myself what aspects of it aren’t well optimized. A lot of it actually comes down to things like email processing, and time spent for example actually responding to questions. Now, of course, I’ve spent lots of effort to try to structure things so as many questions as possible become self-answering, or can be addressed with technology and automation that we’ve built. And, in my role as CEO, I also try hard to delegate to other people whenever I can.
But there’s still plenty left. And I certainly wonder whether with all the technology we now have, more could be automated, or delegated to machines. Perhaps all that data I’ve collected on myself will one day let one basically just built a “bot of me”. Having seen so many of my emails—and being able to look at all my files and personal analytics—maybe it’s actually possible to predict how I’d respond to any particular question.
We’re not there yet. But it will be an interesting moment when a machine can, for example, have three ideas about how to respond to something, and then show me drafts that I can just pick from and approve. The overall question of what direction I want to go in will almost by definition have to stay with me, but the details of how to get there I’m hoping can increasingly be automated.