...did Kubla Khan a stately pleasure-dome decree...



  • Being pretty heavy on the ADD myself, I started writing this earlier today. I thought I would post what I have done of it so far, simply because I will probably forget about it for a few months or years before I get around to working on it some more.

    Welcome to the Pleasuredome
    A history of the longest-running software project ever attempted

    It all started with a boy, and a problem.

    Theodor Holm Nelson was the child of two B-list Hollywood actors, being raised by his grandparents in Chicago at a time when technology seemed both limitless and terrifying. Ted was the sort of bright, restless kid who seems to pop up from time to time, brilliant but disjointed, the type who seemed destined to become the butt of absent-minded professor jokes later in life (a destiny he would come to fulfill in the 1990s during his tenure at Keio University). While today such a child would be loaded down with Adderall and have his social interactions carefully monitored, in the 1950s no one had heard of ADD or Asperger's Syndrome, so he was simply dismissed as 'odd' and left to his own devices. When his family later moved to Greenwich Village, the quirky teen was a perfect fit for the quirky New York City neighborhood.

    He had hundreds of ideas running around his head all the time, weird connections no one else seemed to make. They almost drove him crazy at times, because he couldn't keep track of them all, and would get frustrated as what he thought were possibly world-shaking concepts evaporated from his brow before he could even get a good look at them.

    By the time he had reached college, he had invented several techniques for coping with this mess, which enabled to him to get by in everyday life, and even flourish. The most important of these was a system of small, hand-made pockets made from manila paper which he would stick on his belt, that he used to keep track of the scraps of paper he was constantly writing notes down on. At the end of the day he would try to organize these into some meaningful order, but found himself just writing more notes to indicate which ideas he thought were connected to which other ones. The reference notes quickly started to outpace the new ideas themselves, but he persisted in this; it was the only thing he had found that came close to keeping this witch's brew of ideas in check.

    Along the way, he devoured hundreds of books and magazine articles, penned novels, experimented with new forms of non-linear poetry, started developing a philosophical system he grandiosely called General Systematics, and wrote what he would later call the first rock opera, more than a decade and a half before Pete Townsend came up with the idea for "Tommy". He was eclectic and knowledgeable, but his inability to stay focused meant that it probably was all for naught.

    And it almost certainly would have been, if it weren't for a little bit of fate in the form of a minor editorial piece Vannavar Bush wrote for Atlantic Monthly on the subject of organizing information. "As We May Think" was, for the most part, a typical piece of post-war technological optimism mixed with equally typical post-war technological apologism, almost as if Bush - a key player in the FDR administration's management of wartime engineering work, including RADAR and the atomic bomb - was trying to convince himself that the engineering he had so long strove to master was good for more than just making better killing machines. It was a throw-away article to most people, but for some - including Ted Nelson - its core idea of using automation to allow for a richer method of indexing and connecting information was a revelation.

    Still, this would be nothing but one more intriguing idea, if Harvard University hadn't offered an introductory course in using their shiny new campus mainframe computer to grad students in the Fall of 1960, which also included working with a much smaller computer as well - the Lincoln Labs TX-0 in MIT's offices at Harvard Square, which unknown to all at the time was poised to become the cradle of hackerdom. Despite the fact that the BA he had earned the year before at Swarthmore College was in Film Direction, and that he had no technical or mathematical background to speak of, on a whim - as usual - Nelson decided to check it out.

    And was thunderstruck.

    While Bush's article - which had informed Ted's system for keeping track of ideas, and so was always at the forefront of his mind - spoke of using microfiche and a mechanical system of pins and wires to track connections in his proposed Memex machine, Ted quickly realized that a computer could do all of this just with a program and a set of tape drives. While he had no idea how difficult this would prove to be, or how limited even the largest computers of the time were, he could project forward to a time when there would be a terminal - or even a whole tiny computer, talking to other computers around the world - on every writer's desk. The idea that this could save him from his own mind - maybe even save the human race from its own ignorance, he thought - became an obsession.

    He named his program 'Xanadu', after the unfinished poem by Samuel Coleridge. The irony of this name would become tragic for him in years to come.



  • I thought it was a @Nagesh topic. The avatars don't help either.


  • area_pol

    There is a demo implementation of that idea on modern technology:


  • Winner of the 2016 Presidential Election

    @ScholRLEA Has anyone ever told you that some of your posts are almost as hard to read as your favorite programming language? 🛂



  • @asdf Oddly enough, no.

    How so? I was deliberately being a bit florid (purple, even), but I didn't think it was unclear.


  • Winner of the 2016 Presidential Election

    @ScholRLEA They're rarely unclear, but they usually require more brain capacity than I'm using while browsing this forum to be understood.



  • @asdf I'm not sure what answer I expected, but that wasn't it. Uh, thanks, I think?

    I'll admit that I partially patterned this on the style Steven Levy used in Hackers, which was in turn cribbed partially from Studs Terkel and Tom Wolfe, I think.



  • @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Along the way, he devoured hundreds of books and magazine articles, penned novels, experimented with new forms of non-linear poetry, started developing a philosophical system he grandiosely called General Systematics, and wrote what he would later call the first rock opera, more than a decade and a half before Pete Townsend came up with the idea for "Tommy".

    Although "Jesus Christ, Superstar" and "Godspell" might not have used the term "rock opera", they were much closer to operas than to regular musicals, because all of the dialog was sung. A standard musical is mostly spoken dialog, with a few songs thrown in for good measure. Wonder if there's any merit to the idea that Nelson's unpublished "rock opera" was more of a rock opera than the two above.


  • Discourse touched me in a no-no place

    @asdf said in ...did Kubla Khan a stately pleasure-dome decree...:

    some of your posts are almost as hard to read as your favorite programming language?

    His favorite programming language isn't really hard to read if you use the right text editor. 😉



  • I came here expecting Rush songs and left disappointed.



  • @mott555 But, but... Welcome to the Pleasuredome was by Frankie Goes to Hollywood, not Rush!

    Now I need to decide if I should post 'Xanadu' (the studio version from A Farewell to Kings, not the Exit Stage Left live recording), "Relax", or that crappy Olivia-Newton John thing to the SotD thread. Since I too am a Rush fan (wow, Schol is into prog rock, what a surprise), I'll probably go with the first one.

    EDIT: Never mind, got Hanzo'ed by RaceProUK


  • Impossible Mission Players - A

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    got Hanzo'ed by RaceProUK

    The trouble with deleting posts.... (I see no evidence of this, 'less it was cross thread...)


  • SockDev

    @Tsaukpaetra I posted 2112 in the Song of the Day thread



  • And TIL that in 1977, Geddy Lee looked an awful lot like Sean Penn as Jeff Spicoli.

    Rush - Xanadu – 11:10
    — RushVEVO

    /me feels sad that I can't grow my hair out like that anymore - thanks a lot, Grandpa Flint



  • This is part of an early draft of a white paper I am writing, because inspirational or not, Ted can't explain his ideas worth shit and most people only hear third- or fourth-hand accounts of them (if that) anyway. If I ever get back to writing "Welcome to the Pleasuredome", I will probably include an abbreviated version of this in it.

    And yeah, I know that the bullet point headers are rather twee and corny, but they are meant to catch the readers attention. Finally, this is just a very early form of this, and I was hoping for some meaningful feedback on it (Ha!) before proceeding any further, but there are still some things I want to have in this list that I am still trying to formulate my explanation for.

    While the philosophy behind Xanadu is complex and nuanced, and has never been fully expressed by Nelson, the key insights can be summarized as:

    • Everything is Deeply Intertwingled - Individual data do not exist in a vacuum, especially when a given datum represents a concept rather than a fact. The connections between ideas is often as important as the ideas themselves, so storage of data should keep track of those connections whenever possible, and allow new connections to be made without changing the data themselves.

    • Data are Forever - Losing data is unacceptable and must be avoided whenever possible. It is important to be able to save ideas indefinitely, because you have no way of knowing which ideas will prove important later. Changes should not overwrite the existing stored data, but rather the new data should be stored as a new trace, and an alternate set of connections should be created while still allowing the existing data and connections to be accessed,; that is to say, changes are stored as deltas of the original.

    • Address Globally, View Locally – All data should have a unique identifier or address that refers to it in all cases. Data should never be explicitly copied, and should never be permanently duplicated at another address. Copying should be used solely for the purposes of caching and data preservation, and all copies should be transparently referenced using the datum's unique address. All copies should be managed by the xanalogical storage, transparent to the client programs.

    • View What Thou Wilt – storage and presentation are orthogonal topics, and the storage of data should not reflect unrelated concerns for how the data can be viewed. It should be possible for a program to fetch and present data in any way it requires without changing the original data.

    • Infinite Diversity in Infinite Combinations – all data must be first-class objects, and it should be possible to combine or associate any arbitrary data that the program has access to.

    • Slice and Dice – Most data are not atomic. It should be possible to access any part of a stored datum independently, unless the format of the datum makes it impossible, without having to access any other parts that aren't immediately needed. Formats for images, sounds, etc. that allow partial retrieval should be favored whenever feasible.

    • Do the Evolution - The evolution of a concept is important, so the path by which an idea was conceived and further developed should be retained to as great a degree as possible.

    • Citation Needed - The source of a datum is relevant, and should always be accessible when it is known. Attribution of ideas is critical to understanding both meaning and significance.


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    I know that the bullet point headers are rather twee and corny

    I completely disagree. For me, lists give the text structure, and make it much easier to quickly understand than a uniform, flowing text. Any notes or texts I write are always nested lists.

    While the philosophy behind Xanadu is complex and nuanced

    Maybe it was complex and nuanced in the ancient times when the project started, now they seem like a natural part of database design, a theoretical ideal which, due to practical reasons, is not usually possible/desirable in real systems.
    For example, immutability is fashionable in functional programming, deduplication is an important goal of database design.

    It is important to be able to save ideas indefinitely, because you have no way of knowing which ideas will prove important later.

    The problem of storage seems to be solved now. The challenge right now is to find the valuable parts in this vast data pool.


  • Discourse touched me in a no-no place

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    The challenge right now is to find the valuable parts in this vast data pool.

    The hard part is ensuring that the critical information that links data to its meaning and context is retained. It's ever so easy to lose that; very few people try to systematically capture it in the first place, so the info that they do remember to keep often becomes disconnected. (The filename is not enough. Really.) However, it tends to be easier for programmers in this area than other disciplines, as there are already systems that can capture the state of the code, what that code was talking about, and the state of the associated information (e.g., bug database) so that you can trace what was really going on. Doing this for many other disciplines is so much more difficult…

    Bulk storage is mostly solved. The consequences for retrieval are a whole 'nother thing.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    I know that the bullet point headers are rather twee and corny

    I completely disagree. For me, lists give the text structure, and make it much easier to quickly understand than a uniform, flowing text. Any notes or texts I write are always nested lists.

    I actually was referring to the specific names I have given the bullet points, rather than the fact I had named them. They are very sound-bite-y names that, while potentially compelling, are also potentially misleading, and honestly aren't the most professional sounding headers. While I personally like them, and I am not going to weep if it doesn't come across as Serious Business in a forum like this one, it is something I will need to consider if I ever get to the point of publishing this material in a formal paper.

    While the philosophy behind Xanadu is complex and nuanced

    Maybe it was complex and nuanced in the ancient times when the project started, now they seem like a natural part of database design, a theoretical ideal which, due to practical reasons, is not usually possible/desirable in real systems.
    For example, immutability is fashionable in functional programming, deduplication is an important goal of database design.

    Good point, although I would also say that this indicates that I am still not making things clear enough yet. Many of the things that were in Xanadu were either known to researchers at the time but hadn't gotten around to practicing coders yet, or conversely were the secret sauce some programmers but hadn't made it to the literature, or else were the sort of thing that someone else was going to re-discover eventually anyway. The thing is, Nelson not only was the first to put a lot of these things out in public, but also saw (and still sees) it as a whole rather than as a bag of techniques - while many if not most of the ideas that are part of Xanadu are found elsewhere today, they generally are only found in specific areas such as database design, or in a very mangled or limited form such as WWW hyperlinks.

    The comparison to databases is an insightful one, as it shows exactly the problem he has had getting this across to people - the whole point of Xanadu is to stop thinking of 'text files', 'database records', 'spreadsheet cells', and so on as unrelated things, but rather have a setup that let's the program work with data in a fine-grained manner - where the data used in a 'text document' really would be the same stored information as the data in a 'spreadsheet', not because of some kludge that let's you insert from one file to another, but because the 'text document' and the 'spreadsheet' are just two of the possible views on data that doesn't have a specific format at all (or at least not one related to a particular application).

    The original idea was to facilitate hyperliterature, for both fiction writing and academic research, where a citation of a paper (for example) could automatically display a sub-window showing the cited work, or insert a quote from the work seamlessly without having to copy the quoted text/images/audio/whatever. He also wanted to have a thorough record of how a story or paper was developed during the process of writing, and allow (to a degree) unlimited backtracking - you would be able to see the whole editing trace, down to at least the paragraph or sentence level, and get a window into the writer's thought processes.

    Eventually, though, it evolved into something simpler when he - along with Mark $amuel Miller (he insists on using the dollar sign, or at least he used to), Stu Greene, Roger Gregory, and the rest, with some kibbitzing from Eric Drexler and Doug Engelbart - generalized the idea, and found that doing this actually made the whole thing come together a lot better.

    Here's my point: what I've put in these bullet points so far are still kind of a translation of what Xanadu is all about, because while the ideas are simple, the way he is using them is really unfamiliar to most people. I hate to use a term as tired and misused as 'paradigm shift', but that's sort of what it takes to see why this is a good idea, and unfortunately, Ted is really terrible at explaining himself - he mostly ends up just cheerleading for a team that isn't even on the field yet, and while that has managed to get support from a handful of people who actually understood what he was saying, it has led to a lot of confusion as well. The whole point of writing this is to try and explain the ideas to people, including, I'm sorry to say, myself.

    The problem of storage seems to be solved now. The challenge right now is to find the valuable parts in this vast data pool.

    Agreed. This was always one of the main points in Xanadu, actually, though I don't think they ever solved it as well as they thought they had.

    I'm going to try and work on the actual explanation some more. While I probably will need more bullet points later, those are just the scaffolding for what I mean to write, anyway. My goal isn't to try to promote Xanadu - that horse left the barn long ago - but to get the ideas behind it out to others who might find them useful, as well as to clarify them for myself in the hope that I will someday have to focus and self-confidence to use them in my own projects.

    BTW, if you ever notice me falling into the "you just don't get it" bullshit, please slap me out of it. If I am trying to explain something to people I know are intelligent and inquisitive, and I can't get the message across, that's means either I am incompetent at explaining the subject, or the ideas themselves are crap. While it is indeed possible I am deluded about xanalogical storage, until I have reason to think that's the case I need to admit that the burden of communication lies on my own shoulders.



  • @ScholRLEA It is an interesting concept, but I do see two fundamental problems with the premise, as demonstrated by the internet:

    1. Links between unrelated data aren't all they're cracked up to be. The idea of hyperlinked literature has pretty much come and gone (I remember when it was a real thing). Quite simply, given the limited information throughput of the human brain, data is best fed in atomic chunks. In the worst cases, too many links lead you down a wiki walk that takes you so far away from your original point of interest that you quite forget how you got here in the first place.

    2. Too much data is the same as no data at all. Having a limited time to acquaint yourself with data, you desperately need a filtering mechanism for "just the good stuff". The problem here is that pretty much the only way to determine what constitutes "the good stuff" is to check for yourself. However, the "limited time" bit, means you will only be able to appraise a minute fraction of everything out there. You must therefore rely on external filters (for example, input from other people), which tends to promote a tiny subset of all available data. The rest may as well not exist. Incidentally, popularity of particular data need have no association with its quality.

    I don't see a convincing technical solution to these problems, because first you would have to formalize fuzzy concepts like "just the good stuff".


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Many of the things that were in Xanadu were either known to researchers at the time but hadn't gotten around to practicing coders yet

    And it still seems to be in that state.
    The concepts on your list feel like an academic proposal of goals we might want to explore, to see how they work in practice.

    The concepts indeed look nice, but I wonder if anyone tried to create example documents in that mixed-embedding format. What kind of documents would be suitable for this? What would be the experience of reading them?

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    but to get the ideas behind it out to others who might find them useful

    It would indeed be valuable to analyse these ideas in the context of our current technology state, existing services and potential use cases.
    That kind of "repository of connected documents" is similar to wikipedia.
    The concept of embedding parts of documents could be useful in documentation and tutorial writing (embed documentation and other helpful resources into a tutorial).


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.