...did Kubla Khan a stately pleasure-dome decree...



  • Being pretty heavy on the ADD myself, I started writing this earlier today. I thought I would post what I have done of it so far, simply because I will probably forget about it for a few months or years before I get around to working on it some more.

    Welcome to the Pleasuredome
    A history of the longest-running software project ever attempted

    It all started with a boy, and a problem.

    Theodor Holm Nelson was the child of two B-list Hollywood actors, being raised by his grandparents in Chicago at a time when technology seemed both limitless and terrifying. Ted was the sort of bright, restless kid who seems to pop up from time to time, brilliant but disjointed, the type who seemed destined to become the butt of absent-minded professor jokes later in life (a destiny he would come to fulfill in the 1990s during his tenure at Keio University). While today such a child would be loaded down with Adderall and have his social interactions carefully monitored, in the 1950s no one had heard of ADD or Asperger's Syndrome, so he was simply dismissed as 'odd' and left to his own devices. When his family later moved to Greenwich Village, the quirky teen was a perfect fit for the quirky New York City neighborhood.

    He had hundreds of ideas running around his head all the time, weird connections no one else seemed to make. They almost drove him crazy at times, because he couldn't keep track of them all, and would get frustrated as what he thought were possibly world-shaking concepts evaporated from his brow before he could even get a good look at them.

    By the time he had reached college, he had invented several techniques for coping with this mess, which enabled to him to get by in everyday life, and even flourish. The most important of these was a system of small, hand-made pockets made from manila paper which he would stick on his belt, that he used to keep track of the scraps of paper he was constantly writing notes down on. At the end of the day he would try to organize these into some meaningful order, but found himself just writing more notes to indicate which ideas he thought were connected to which other ones. The reference notes quickly started to outpace the new ideas themselves, but he persisted in this; it was the only thing he had found that came close to keeping this witch's brew of ideas in check.

    Along the way, he devoured hundreds of books and magazine articles, penned novels, experimented with new forms of non-linear poetry, started developing a philosophical system he grandiosely called General Systematics, and wrote what he would later call the first rock opera, more than a decade and a half before Pete Townsend came up with the idea for "Tommy". He was eclectic and knowledgeable, but his inability to stay focused meant that it probably was all for naught.

    And it almost certainly would have been, if it weren't for a little bit of fate in the form of a minor editorial piece Vannavar Bush wrote for Atlantic Monthly on the subject of organizing information. "As We May Think" was, for the most part, a typical piece of post-war technological optimism mixed with equally typical post-war technological apologism, almost as if Bush - a key player in the FDR administration's management of wartime engineering work, including RADAR and the atomic bomb - was trying to convince himself that the engineering he had so long strove to master was good for more than just making better killing machines. It was a throw-away article to most people, but for some - including Ted Nelson - its core idea of using automation to allow for a richer method of indexing and connecting information was a revelation.

    Still, this would be nothing but one more intriguing idea, if Harvard University hadn't offered an introductory course in using their shiny new campus mainframe computer to grad students in the Fall of 1960, which also included working with a much smaller computer as well - the Lincoln Labs TX-0 in MIT's offices at Harvard Square, which unknown to all at the time was poised to become the cradle of hackerdom. Despite the fact that the BA he had earned the year before at Swarthmore College was in Film Direction, and that he had no technical or mathematical background to speak of, on a whim - as usual - Nelson decided to check it out.

    And was thunderstruck.

    While Bush's article - which had informed Ted's system for keeping track of ideas, and so was always at the forefront of his mind - spoke of using microfiche and a mechanical system of pins and wires to track connections in his proposed Memex machine, Ted quickly realized that a computer could do all of this just with a program and a set of tape drives. While he had no idea how difficult this would prove to be, or how limited even the largest computers of the time were, he could project forward to a time when there would be a terminal - or even a whole tiny computer, talking to other computers around the world - on every writer's desk. The idea that this could save him from his own mind - maybe even save the human race from its own ignorance, he thought - became an obsession.

    He named his program 'Xanadu', after the unfinished poem by Samuel Coleridge. The irony of this name would become tragic for him in years to come.



  • I thought it was a @Nagesh topic. The avatars don't help either.


  • area_pol

    There is a demo implementation of that idea on modern technology:


  • Winner of the 2016 Presidential Election

    @ScholRLEA Has anyone ever told you that some of your posts are almost as hard to read as your favorite programming language? 🛂



  • @asdf Oddly enough, no.

    How so? I was deliberately being a bit florid (purple, even), but I didn't think it was unclear.


  • Winner of the 2016 Presidential Election

    @ScholRLEA They're rarely unclear, but they usually require more brain capacity than I'm using while browsing this forum to be understood.



  • @asdf I'm not sure what answer I expected, but that wasn't it. Uh, thanks, I think?

    I'll admit that I partially patterned this on the style Steven Levy used in Hackers, which was in turn cribbed partially from Studs Terkel and Tom Wolfe, I think.



  • @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Along the way, he devoured hundreds of books and magazine articles, penned novels, experimented with new forms of non-linear poetry, started developing a philosophical system he grandiosely called General Systematics, and wrote what he would later call the first rock opera, more than a decade and a half before Pete Townsend came up with the idea for "Tommy".

    Although "Jesus Christ, Superstar" and "Godspell" might not have used the term "rock opera", they were much closer to operas than to regular musicals, because all of the dialog was sung. A standard musical is mostly spoken dialog, with a few songs thrown in for good measure. Wonder if there's any merit to the idea that Nelson's unpublished "rock opera" was more of a rock opera than the two above.


  • BINNED

    @asdf said in ...did Kubla Khan a stately pleasure-dome decree...:

    some of your posts are almost as hard to read as your favorite programming language?

    His favorite programming language isn't really hard to read if you use the right text editor. 😉



  • I came here expecting Rush songs and left disappointed.



  • @mott555 But, but... Welcome to the Pleasuredome was by Frankie Goes to Hollywood, not Rush!

    Now I need to decide if I should post 'Xanadu' (the studio version from A Farewell to Kings, not the Exit Stage Left live recording), "Relax", or that crappy Olivia-Newton John thing to the SotD thread. Since I too am a Rush fan (wow, Schol is into prog rock, what a surprise), I'll probably go with the first one.

    EDIT: Never mind, got Hanzo'ed by RaceProUK


  • Notification Spam Recipient

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    got Hanzo'ed by RaceProUK

    The trouble with deleting posts.... (I see no evidence of this, 'less it was cross thread...)


  • FoxDev

    @Tsaukpaetra I posted 2112 in the Song of the Day thread



  • And TIL that in 1977, Geddy Lee looked an awful lot like Sean Penn as Jeff Spicoli.

    https://www.youtube.com/watch?v=SEuOoMprDqg

    /me feels sad that I can't grow my hair out like that anymore - thanks a lot, Grandpa Flint



  • This is part of an early draft of a white paper I am writing, because inspirational or not, Ted can't explain his ideas worth shit and most people only hear third- or fourth-hand accounts of them (if that) anyway. If I ever get back to writing "Welcome to the Pleasuredome", I will probably include an abbreviated version of this in it.

    And yeah, I know that the bullet point headers are rather twee and corny, but they are meant to catch the readers attention. Finally, this is just a very early form of this, and I was hoping for some meaningful feedback on it (Ha!) before proceeding any further, but there are still some things I want to have in this list that I am still trying to formulate my explanation for.

    While the philosophy behind Xanadu is complex and nuanced, and has never been fully expressed by Nelson, the key insights can be summarized as:

    • Everything is Deeply Intertwingled - Individual data do not exist in a vacuum, especially when a given datum represents a concept rather than a fact. The connections between ideas is often as important as the ideas themselves, so storage of data should keep track of those connections whenever possible, and allow new connections to be made without changing the data themselves.

    • Data are Forever - Losing data is unacceptable and must be avoided whenever possible. It is important to be able to save ideas indefinitely, because you have no way of knowing which ideas will prove important later. Changes should not overwrite the existing stored data, but rather the new data should be stored as a new trace, and an alternate set of connections should be created while still allowing the existing data and connections to be accessed,; that is to say, changes are stored as deltas of the original.

    • Address Globally, View Locally – All data should have a unique identifier or address that refers to it in all cases. Data should never be explicitly copied, and should never be permanently duplicated at another address. Copying should be used solely for the purposes of caching and data preservation, and all copies should be transparently referenced using the datum's unique address. All copies should be managed by the xanalogical storage, transparent to the client programs.

    • View What Thou Wilt – storage and presentation are orthogonal topics, and the storage of data should not reflect unrelated concerns for how the data can be viewed. It should be possible for a program to fetch and present data in any way it requires without changing the original data.

    • Infinite Diversity in Infinite Combinations – all data must be first-class objects, and it should be possible to combine or associate any arbitrary data that the program has access to.

    • Slice and Dice – Most data are not atomic. It should be possible to access any part of a stored datum independently, unless the format of the datum makes it impossible, without having to access any other parts that aren't immediately needed. Formats for images, sounds, etc. that allow partial retrieval should be favored whenever feasible.

    • Do the Evolution - The evolution of a concept is important, so the path by which an idea was conceived and further developed should be retained to as great a degree as possible.

    • Citation Needed - The source of a datum is relevant, and should always be accessible when it is known. Attribution of ideas is critical to understanding both meaning and significance.


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    I know that the bullet point headers are rather twee and corny

    I completely disagree. For me, lists give the text structure, and make it much easier to quickly understand than a uniform, flowing text. Any notes or texts I write are always nested lists.

    While the philosophy behind Xanadu is complex and nuanced

    Maybe it was complex and nuanced in the ancient times when the project started, now they seem like a natural part of database design, a theoretical ideal which, due to practical reasons, is not usually possible/desirable in real systems.
    For example, immutability is fashionable in functional programming, deduplication is an important goal of database design.

    It is important to be able to save ideas indefinitely, because you have no way of knowing which ideas will prove important later.

    The problem of storage seems to be solved now. The challenge right now is to find the valuable parts in this vast data pool.


  • Discourse touched me in a no-no place

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    The challenge right now is to find the valuable parts in this vast data pool.

    The hard part is ensuring that the critical information that links data to its meaning and context is retained. It's ever so easy to lose that; very few people try to systematically capture it in the first place, so the info that they do remember to keep often becomes disconnected. (The filename is not enough. Really.) However, it tends to be easier for programmers in this area than other disciplines, as there are already systems that can capture the state of the code, what that code was talking about, and the state of the associated information (e.g., bug database) so that you can trace what was really going on. Doing this for many other disciplines is so much more difficult…

    Bulk storage is mostly solved. The consequences for retrieval are a whole 'nother thing.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    I know that the bullet point headers are rather twee and corny

    I completely disagree. For me, lists give the text structure, and make it much easier to quickly understand than a uniform, flowing text. Any notes or texts I write are always nested lists.

    I actually was referring to the specific names I have given the bullet points, rather than the fact I had named them. They are very sound-bite-y names that, while potentially compelling, are also potentially misleading, and honestly aren't the most professional sounding headers. While I personally like them, and I am not going to weep if it doesn't come across as Serious Business in a forum like this one, it is something I will need to consider if I ever get to the point of publishing this material in a formal paper.

    While the philosophy behind Xanadu is complex and nuanced

    Maybe it was complex and nuanced in the ancient times when the project started, now they seem like a natural part of database design, a theoretical ideal which, due to practical reasons, is not usually possible/desirable in real systems.
    For example, immutability is fashionable in functional programming, deduplication is an important goal of database design.

    Good point, although I would also say that this indicates that I am still not making things clear enough yet. Many of the things that were in Xanadu were either known to researchers at the time but hadn't gotten around to practicing coders yet, or conversely were the secret sauce some programmers but hadn't made it to the literature, or else were the sort of thing that someone else was going to re-discover eventually anyway. The thing is, Nelson not only was the first to put a lot of these things out in public, but also saw (and still sees) it as a whole rather than as a bag of techniques - while many if not most of the ideas that are part of Xanadu are found elsewhere today, they generally are only found in specific areas such as database design, or in a very mangled or limited form such as WWW hyperlinks.

    The comparison to databases is an insightful one, as it shows exactly the problem he has had getting this across to people - the whole point of Xanadu is to stop thinking of 'text files', 'database records', 'spreadsheet cells', and so on as unrelated things, but rather have a setup that let's the program work with data in a fine-grained manner - where the data used in a 'text document' really would be the same stored information as the data in a 'spreadsheet', not because of some kludge that let's you insert from one file to another, but because the 'text document' and the 'spreadsheet' are just two of the possible views on data that doesn't have a specific format at all (or at least not one related to a particular application).

    The original idea was to facilitate hyperliterature, for both fiction writing and academic research, where a citation of a paper (for example) could automatically display a sub-window showing the cited work, or insert a quote from the work seamlessly without having to copy the quoted text/images/audio/whatever. He also wanted to have a thorough record of how a story or paper was developed during the process of writing, and allow (to a degree) unlimited backtracking - you would be able to see the whole editing trace, down to at least the paragraph or sentence level, and get a window into the writer's thought processes.

    Eventually, though, it evolved into something simpler when he - along with Mark $amuel Miller (he insists on using the dollar sign, or at least he used to), Stu Greene, Roger Gregory, and the rest, with some kibbitzing from Eric Drexler and Doug Engelbart - generalized the idea, and found that doing this actually made the whole thing come together a lot better.

    Here's my point: what I've put in these bullet points so far are still kind of a translation of what Xanadu is all about, because while the ideas are simple, the way he is using them is really unfamiliar to most people. I hate to use a term as tired and misused as 'paradigm shift', but that's sort of what it takes to see why this is a good idea, and unfortunately, Ted is really terrible at explaining himself - he mostly ends up just cheerleading for a team that isn't even on the field yet, and while that has managed to get support from a handful of people who actually understood what he was saying, it has led to a lot of confusion as well. The whole point of writing this is to try and explain the ideas to people, including, I'm sorry to say, myself.

    The problem of storage seems to be solved now. The challenge right now is to find the valuable parts in this vast data pool.

    Agreed. This was always one of the main points in Xanadu, actually, though I don't think they ever solved it as well as they thought they had.

    I'm going to try and work on the actual explanation some more. While I probably will need more bullet points later, those are just the scaffolding for what I mean to write, anyway. My goal isn't to try to promote Xanadu - that horse left the barn long ago - but to get the ideas behind it out to others who might find them useful, as well as to clarify them for myself in the hope that I will someday have to focus and self-confidence to use them in my own projects.

    BTW, if you ever notice me falling into the "you just don't get it" bullshit, please slap me out of it. If I am trying to explain something to people I know are intelligent and inquisitive, and I can't get the message across, that's means either I am incompetent at explaining the subject, or the ideas themselves are crap. While it is indeed possible I am deluded about xanalogical storage, until I have reason to think that's the case I need to admit that the burden of communication lies on my own shoulders.


  • Trolleybus Mechanic

    @ScholRLEA It is an interesting concept, but I do see two fundamental problems with the premise, as demonstrated by the internet:

    1. Links between unrelated data aren't all they're cracked up to be. The idea of hyperlinked literature has pretty much come and gone (I remember when it was a real thing). Quite simply, given the limited information throughput of the human brain, data is best fed in atomic chunks. In the worst cases, too many links lead you down a wiki walk that takes you so far away from your original point of interest that you quite forget how you got here in the first place.

    2. Too much data is the same as no data at all. Having a limited time to acquaint yourself with data, you desperately need a filtering mechanism for "just the good stuff". The problem here is that pretty much the only way to determine what constitutes "the good stuff" is to check for yourself. However, the "limited time" bit, means you will only be able to appraise a minute fraction of everything out there. You must therefore rely on external filters (for example, input from other people), which tends to promote a tiny subset of all available data. The rest may as well not exist. Incidentally, popularity of particular data need have no association with its quality.

    I don't see a convincing technical solution to these problems, because first you would have to formalize fuzzy concepts like "just the good stuff".


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Many of the things that were in Xanadu were either known to researchers at the time but hadn't gotten around to practicing coders yet

    And it still seems to be in that state.
    The concepts on your list feel like an academic proposal of goals we might want to explore, to see how they work in practice.

    The concepts indeed look nice, but I wonder if anyone tried to create example documents in that mixed-embedding format. What kind of documents would be suitable for this? What would be the experience of reading them?

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    but to get the ideas behind it out to others who might find them useful

    It would indeed be valuable to analyse these ideas in the context of our current technology state, existing services and potential use cases.
    That kind of "repository of connected documents" is similar to wikipedia.
    The concept of embedding parts of documents could be useful in documentation and tutorial writing (embed documentation and other helpful resources into a tutorial).


  • Discourse touched me in a no-no place

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    The concepts indeed look nice, but I wonder if anyone tried to create example documents in that mixed-embedding format. What kind of documents would be suitable for this? What would be the experience of reading them?

    Arguably that's the web. An interconnected mesh of documents need not be a repository of wisdom; foolishness looks ever so similar at the graph level…


  • area_pol

    @dkf said in ...did Kubla Khan a stately pleasure-dome decree...:

    Arguably that's the web.

    I read some article about Xanadu and it was shown as an opposition to the normal web, as though they have the same role.

    But websites are not patchworks of iframes pulling in parts of random articles.
    Conversely, practice may show the obsolescence of hypertext model. What is the point of putting links in every word in wikipedia? If I want to find an article about something, I can search for it.


  • Discourse touched me in a no-no place

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    What is the point of putting links in every word in wikipedia?

    That never made sense. Linking key concepts works. Words aren't usually the same as concepts despite being related (case in point: the is not normally a word you'd hyperlink by itself, though there are times when it makes sense).

    But that wasn't really my point. My point was that building hyperlinked graphs of knowledge does work, and is something we've actually successfully done, but it doesn't work quite as well as the original proponents of it envisaged. The problem is that as soon as you try to build a consistent graph of the overall state of knowledge of a large chunk of the world (as opposed to just a little bit of it) you end up with a horribly inconsistent where there's a lot of nodes and arcs that are sort of the same but not quite. You also need to make your graph aware of the temporal nature of knowledge, and that adds a lot more complexity. It's messy, and it's messy in ways that most people don't believe (usually because they just work in their own little domain, which simplifies things a lot).



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    What is the point of putting links in every word in wikipedia?

    To give the people who work on the 'disambiguation pages with links' list something to do? I guarantee if someone's trying to link every word, they're just linking it to the page with the same title as the word, or at best the nearest match that's not a redlink.



  • I'm writing up some more stuff to post here, mainly as a reply to some of the comments, but I just found out that someone already wrote the kind of biography I was starting here:

    I haven't read it yet, so I don't know if it is any good or not, but just the fact that it's out there already - from just last year, at that - means I might be wasting my time with "Welcome to the Pleasuredome".


  • Discourse touched me in a no-no place

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    I haven't read it yet, so I don't know if it is any good or not, but just the fact that it's out there already - from just last year, at that - means I might be wasting my time with "Welcome to the Pleasuredome".

    I think I've seen stuff on this general topic in other places too. This isn't an area that is unknown, but rather more of a path only infrequently travelled. The real question would then be whether you'd be revealing anything new (assuming that readers had already read that book) or if you would be creating a presentation more suitable for a wider audience; if so, continuing would definitely be valuable.

    Overall, the big problem with the whole hypermedia approach is that building a good network to install (i.e., a truly rich collection of links between content) is usually at odds with building good content at each of the nodes that you link between. I suspect this is because there's a limited amount of effort overall. It would be interesting to see whether current research into AI will allow more machine assistance in this area.


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    means I might be wasting my time

    I have no idea whether that is in the book, but you could focus on "how could we use these concepts in practice" or actually try making example documents in xanadu format, so that we could see if the format is bearable at all.

    Also, in the time they spent on writing a book, they could implement the project's prototype, which should be easy using modern technology.



  • @dkf said in ...did Kubla Khan a stately pleasure-dome decree...:

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    What is the point of putting links in every word in wikipedia?

    That never made sense. Linking key concepts works. Words aren't usually the same as concepts despite being related (case in point: the is not normally a word you'd hyperlink by itself, though there are times when it makes sense).

    Comparing Xanadu to the WWW is like comparing NTFS to MS Word. Xanalogical storage isn't about displaying a page with inline links to other pages; while that is certainly one application of it, and one of the original goals of the project, that's just one use of a far more general structure - it's a front end, use presentation concern, not the kind of system-level concern Xanadu is about.

    Just to clarify this point: xanalogical links aren't much like HTML hyperlinks at all. Xanadu isn't a replacement browser or hypertext format; it meant to supplant the idea of file systems. The hypermedia are what it is meant to support, but it isn't the hypermedia themselves (except in the sense of a virtual storage medium).

    Xanalogical links are data structures which are separate from the more general sorts of data they are used to index and manipulate. There are several types of links, most of which the user would never interact with directly; speaking of having a default presentation of Xanadu links makes no more sense than speaking of having a default presentation of EXT4 i-nodes (the i-nodes themselves, that is, not the file data they point to).



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    means I might be wasting my time

    I have no idea whether that is in the book, but you could focus on "how could we use these concepts in practice" or actually try making example documents in xanadu format, so that we could see if the format is bearable at all.

    One of the things I am working is a reference implementation, actually. Whether I can focus on it long enough to get anywhere remains to be seen.

    Also, in the time they spent on writing a book, they could implement the project's prototype, which should be easy using modern technology.

    True enough. I have exactly that in mind myself, with both the papers I am writing, and the biography I started here, being mainly to help me focus and clarify the my own thoughts on the subject.

    While a 'working' Xanadu system is available, it is really not a working system much at all, and I don't see it becoming one. There are several partial implementations around, such as Alph, Dolphin, Tent, and Fenfire, and Ted has used parts of it in ZigZag and Cosmic Book, but the 'official' Xanadu site is a shadow of what it was supposed to be.


  • Discourse touched me in a no-no place

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Xanalogical links are data structures which are separate from the more general sorts of data they are used to index and manipulate.

    You ought to check out things like the Semantic Web and graph databases; they sound much more like what you're talking about and it's where there's quite a lot of current research work into their applications. And the real problems come when you have multiple conflicting sources of truth; it's hard to make a model that doesn't either implode under its own weight in the face of such stupidity or conclude that black is white (before getting itself killed on the next pedestrian crossing. :))



  • @dkf Thanks for the reminder; I keep meaning to look into both of those for a while, but keep forgetting about it shortly deciding to. This would probably be a good time to get to that.


  • Discourse touched me in a no-no place

    @ScholRLEA OK, some potted information.

    Semantic Web: started out as way of “letting computers know what the links on web pages really mean” though it's deviated somewhat. The big outcome from this is RDF, which is a way of describing a graph with links. Yes, it's URL heavy, but think of URLs as just being general identifiers (that you can possibly resolve to something to read, you human meatbag you!). There's also more sophisticated levels of reasoning in places (e.g., many variations on OWL), but most people try to disable that because it usually makes everything intractable.

    Graph databases: partially coming from the need to understand these RDF-described graphs, there are now software systems to allow you to create non-trivial queries over a graph of heterogenous links. There's also systems to let you visualise the results. We're investigating the use of this in biochemistry quite a bit, where there's a crapload of data and Going Full Relational (never go full relational!) involves doing joins out the wazoo. In fact, the main advantage over normal relational DBs that graph DBs have is that they're much better at what would be joins; it's the non-join parts that they tend to be weaker with.

    We've built some pretty big graph databases, but currently they're mostly rather proprietary due to the way that the projects to do the building were funded. They also require you to know a lot of chemistry to get good answers out of. ;)



  • I'm working on translating the existing Xanadu glossary from Xanadu-Speak into plain(er) English, but while trying to define tumblers, I got to a point where I needed more of an explanation of the idea than would fit into a short definition. I'll keep the first sentence for the glossary, but I want to at least try to thrash out some aspects of it first.

    Tumblers are a category of transfinite numbers originally designed by Roger Gregory for the Xu.78.1 addressing system. While they are used for addressing and identification (more on the difference later), they are pretty much a fully-realized numerical subspace unto themselves, and Gregory worked out the basics of tumbler arithmetic in the mid-1970s.

    As best as I can determine, a tumbler is basically a sparse jagged matrix consisting of a series of vectors separated by null values, with each vector consisting of one or more integer values. This peculiar structure comes from the intention to having the node (machine) addresses be composable from the network heirarchy, in a fashion similar to that used later by Appletalk. While they are matrices, they are usually written in a linear fashion rather misleadingly resembling an IP4 address. For example,

    1.0.1024.48.17.17779.0.56.12.3222.5.0.2.67.0
    

    which might alternatively (and probably more accurately) be given as

    [   1               ]
    [1024 48    17 17779]
    [  56 12  3222     5]
    [   2 67            ]
    

    The tumbler addresses described publicly start with a leading start marker (a byte of value 1) and use the successive integer values as sections of the address, with the numbers consisting of one or more non-zero bytes and the nulls represented by zeros.

    1. < node >.0. < user >.0. < document >.0. < element >
    

    The basic sections are the node address; the creator's network ID (based on the node their account was created on); a document ID or root address; a data format ID; and optionally, a three-part span giving an offset into the address, a direction marker, and the size (or granularity, anyway) of the section being addressed.

    Now, the astute will notice two things. First off, these aren't easily human-readable, even in the dot notation given. This supposedly was intentional, as the tumblers were supposed to be entirely for the use of the system, not the users or even the programmers; most of the Xanadu folks even argue that user-accessible addresses are a Bad Thing, as they not only can be entered incorrectly, they potentially can be used to subvert the security, publication management (i.e, DRM), and versioning systems. How much this is really something they thought out ahead of time, and how much is sour grapes over the success of HTML and URIs, I have no idea.

    Second, the 'addresses' as given are really addresses at all (not even virtual ones), but identifiers; a tumbler address of this type gives no way of determining the actual physical location of the data, and more importantly, the same address is used to identify the data regardless of where it is actually stored, including cached or mirrored versions, even when the machine it was created on has been decommissioned. I have some inkling about how these logical addresses are mapped to another layer of tumblers that serve as a kinda-sorta physical address, but both the documentation and code currently released are, to borrow a phrase from Yahtzee, "as twisted and impenetrable as a granite octopus".

    Getting to the bottom of this business is going to be a lot of work, and I am not exactly known for sticking the landing, so to speak. I want to understand it, but I'm not sure I can.



  • On another tangent, I should explain that while Ted Nelson did coin the term 'hyperlink', he and the Xanadu team had a habit of using the word 'link' as a catch-all term for any kind of meta-data attached to a particular datum. While this is not so bizarre as it may sound - the term 'meta-data' wasn't really in use much yet (according to Wicked-Pedo, it was first used by Phil Bagley in 1968), and they certainly started out using it primarily for referring to out-of-band hyperlinks, it just sort of got generalized somewhere along the way and they never bothered to change how they used it unless they were being really specific - it really makes it hard to explain what they are talking about sometimes. Unfortunately, I just noticed that I have been doing the same thing, hence the need to explain it now.


  • area_pol

    @ScholRLEA This could be summarized as:
    A resource is identified by a unique primary key of (service, user_account, document_id, document_offset).

    Instead of matrices, byte markers etc, just use a standard DB. Or one of the many distributed storage systems that are now published every week.

    While the proposed features of xanadu may be interesting, I would not explore their technical implementation. After all, their implementation efforts were a total failure.

    most of the Xanadu folks even argue that user-accessible addresses are a Bad Thing, as they not only can be entered incorrectly, they potentially can be used to subvert the security

    Hiding URLs is just obscurity, not security.



  • @Adynathos Good point, actually. I have been thinking of dropping tumblers from the reference implementation (which would be more of working demonstrator of the ideas rather than a realistic implementation) if I ever get it done at all. They are sort of a 1970s view of addresses and meta-data, to be honest, and I don't really think they are as practical as all that.

    However, if I am to have any hope of distilling the actual good ideas out this stew of ideas, I need to understand the whole thing on at least a general level.

    Also, you have to understand that several of the goals and implementation techniques of Xanadu were very similar to those of a database engine - Ted did try to write a 'Xanadu Lite' using an RDBMS back end around 1999, in fact, and decided that that the database engine basically was the same thing that he'd been trying to do, but less general. Make of that what you will.

    As for the implementations failing, realistically, the failure was not in the technology but in the lunacy surrounding the development. None of them got developed to the point where they could fail, even in the way something like Big Rigs Racing did - the process was so broken they never had a chance to see if the technology was any good.


  • Trolleybus Mechanic

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    First off, these aren't easily human-readable, even in the dot notation given. This supposedly was intentional, as the tumblers were supposed to be entirely for the use of the system, not the users or even the programmers; most of the Xanadu folks even argue that user-accessible addresses are a Bad Thing, as they not only can be entered incorrectly, they potentially can be used to subvert the security, publication management (i.e, DRM), and versioning systems.

    That is a pretty sensible approach. I've always been a fan of separating machine-readable identifiers from human-readable ones. Nobody really has any business caring about how the computer identifies a certain piece of data, provided the identification is unambiguous and accurate. Humans are much better equipped to deal with fuzziness than computers, but really don't have the headspace to deal with a huge address-space.

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    Hiding URLs is just obscurity, not security.

    One man's output is another man's input. One reason to keep humans away from the address space is to keep them from poisoning the well (also known as spoofing). Based on what @Schol_R_LEA wrote - presuming it reflects the views of the authors - the issue seems less a matter of user-readable addresses, but user-provided addresses.



  • That's part of it, yes, but it also highlights the real difference regarding what Tim Berners-Lee called a 'hyperlink' and what Nelson & Co. had in mind. Even setting aside the "links == metadata" issue, the links they were talking about weren't directly visible to the user at all - presenting them, or not presenting them, was entirely on the application's shoulders. There was no equivalent of a browser address bar in Xanadu because there was no equivalent of an overarching browser in the WWW sense, period. Xanalogical storage is just one back end part of a larger system, one designed to make hypermedia seamless across several different applications without needing some intermediary like OLE to frankenstein the documents together.

    When I say that Xanadu was mostly about getting rid of the idea of discrete files, I really mean it. Ted really sees things this way, as a cloud of interconnected ideas that reaches out into infinity, constantly expanding, merging, and dissolving, and he can't understand why most other people don't (actually, he - like his inspiration, Vannevar Bush - thinks most people do think this way, but are too afraid to look at raw data without some kind of hierarchical blinders). The Xanadu 'docuverse' reflects his whole way of thinking.

    Hyperlinks of the kind that a user can click on were just one small part of what this was about; most hyperlinks were to be used to stitch parts of different documents together into a new view, rather than as something the user would see, or to facilitate comparisons between documents, or even serve as a placeholder for incoming data streams such as stock prices during the trading day. I think that what pisses Ted off more than anything is that HTML took one trivial part of his ideas and threw away almost everything else, in a way that made most people think that this was all there was to his work.

    Well, that, and the continued existence of IBM, but that's only because he still remembers with bitterness the days when most computer users marched along the One True Path of Glorious Batch-Processed Socialism that Thomas Watson Jr. had set them on. Both the original 1974 version of Computer Lib and the 1987 update had page after page of screeds about the evils of Big Blue; from some of the things he's said more recently, I think he still hates them even more than he does Microsoft or Oracle, and that's saying something.


  • Discourse touched me in a no-no place

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    Hyperlinks of the kind that a user can click on were just one small part of what this was about; most hyperlinks were to be used to stitch parts of different documents together into a new view, rather than as something the user would see, or to facilitate comparisons between documents, or even serve as a placeholder for incoming data streams such as stock prices during the trading day. I think that what pisses Ted off more than anything is that HTML took one trivial part of his ideas and threw away almost everything else, in a way that made most people think that this was all there was to his work.

    The silly thing about that is that people are doing that sort of thing now.

    Without a virtual addressing scheme that puts OIDs to shame.



  • @dkf said in ...did Kubla Khan a stately pleasure-dome decree...:

    The silly thing about that is that people are doing that sort of thing now.

    "Why should car manufacturers put bumpers on their cars? We did fine for years without them, and if a driver thinks he needs one, he can always just weld a piece of sheet metal to the front."

    "Why should Bell telephone put up a bunch of pre-laid cables from place to place and have something as complicated as switchboards for people to call through? We are doing just fine by laying cables from one place to another whenever we expect to need telephone service later."

    "Who needs a security system in their operating system? A sensible programmer can always write their own virus scanner as a part of their application."

    Just because some of the things Xanadu was designed from the ground up to do can now be done in a half-assed manner by layering a ton of extra crap onto HTML doesn't mean that Xanadu was a bad idea. If anything, the fact that people are kludging these features onto systems that don't have them shows that they were a good idea all along.

    Whether they were practical from the start is another matter. I doubt that Xanadu would be getting even as much attention as it is now if Tim Berners-Lee hadn't read Literary Machines and said, "Hey, I bet I can do something that does just enough of this stuff to be useful as a way of sharing research papers before publication." Part of why Xanadu failed was because they were too ambitious, trying to swallow the whole elephant at once without enough funding (even with Autodesk behind them) or oversight for how the development was going. The ideas were good, but they wouldn't be usable with the hardware of the day, even if they hadn't completely fucked up the development process - trying to get this off the ground with 386s, 40M hard drives, and dial-up connections was like trying to drive a Ferrari across Siberia in 1800.

    Hell, at one point in 1980, their development system was a leased Data General mini with a 10M drive the size of a dorm fridge, and they lost the whole codebase (which I gather was written in FORTRAN, of all things) when the lease expired. And that's not even considering that, even then, this guru of small computers was aiming at a system running on a network of mainframes that would serve a string of franchised 'terminal centers' (basically what would be an internet cafe today) modeled after McDonald's.

    Without a virtual addressing scheme that puts OIDs to shame.

    I'm not going to defend tumbler addressing, at least not until I have a better idea of why they decided they needed something so complicated. As I said, it was sort of a 1970s idea of how to do addressing in a flexible and scalable way, and I think it was a 'it seemed like a good idea at the time' thing that got superseded when better approaches were developed. Even Mark $ Miller, who worked on developing the idea, has since said that it wasn't really adequate or practical (though not surprisingly, his main beef with it was that it was still too visible, and thus couldn't be used with the kind of capabilities-based security that have been the main focus of his career since the early 1990s).

    People who have some good ideas also tend to have tons of crappy ideas, and Ted is sort of the Ron Popeil of computing in that way - he's endlessly coming up with new ideas, but most of them don't even get out of his mouth before he realizes they are wacky. Considering how many wacky ideas he does express, that's saying something. Despite his talk of 'butterfly mind' being his major strength, there's a reason why ADD and Asperger's Syndrome are considered illnesses.

    Trust me on this, I know. I don't think I have AS (I haven't been diagnosed, at least), but I definitely have a certain amount of ADD. Ted? In person he comes across like he's just taken a low-dose mix of acid and crystal meth. He's brilliant, in his own way, but keeping up with him is a challenge. Realizing that Xanadu was to be his coping mechanism for this goes a long way in explaining both the structure and the history of the system.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    I wonder if anyone tried to create example documents in that mixed-embedding format

    Some bits of it happened in NewtonOS. OLE and OpenDoc both did a certain amount of this as well, although in a very clunky manner.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    The concepts indeed look nice, but I wonder if anyone tried to create example documents in that mixed-embedding format.

    Does the name Marc Andreessen mean anything to you? I mean, yeah, as I keep saying, the fact that HTML <img> tags are inline is sort of bogus from the Xanadu perspective, but they are still a pretty basic example of that. The real difference come from the fact that they, and the hyperlinks, are still kind of stuck in the amber of the HTML code, and it takes a bit of hackery to make something like a piece of text from one document appear seamlessly within another document without copying the shared text into the second document by cut and paste.


  • area_pol

    @ScholRLEA Heh, indeed, with <img>s we use this embedded document system and don't even notice it.

    It would be interesting to see how this works for texts though.
    The text would need a common format (images already use a set of common formats). One may say that HTML is such format, but if we want more than basic formatting like <p> <b> <i>, mixing various HTMLs with various stylesheets will result in chaos.
    And some advanced formatting is needed for code, documentation, equations.



  • DiscoMarkdown! 🎛


  • Discourse touched me in a no-no place

    @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    One may say that HTML is such format

    Indeed it is. <iframe> gives you the capability.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    @ScholRLEA Heh, indeed, with <img>s we use this embedded document system and don't even notice it.

    It would be interesting to see how this works for texts though.
    The text would need a common format (images already use a set of common formats). One may say that HTML is such format, but if we want more than basic formatting like <p> <b> <i>, mixing various HTMLs with various stylesheets will result in chaos.
    And some advanced formatting is needed for code, documentation, equations.

    That touches on some of the problems fans of Xanadu have with HTML and HTTP in general:

    • Links are inline, rather than meta-data, which means that they cannot be manipulated programmatically without at least fetching the HTML file itself.

    • Because the documents consist of a mix of text and tags, with the tags controlling everything except the text, text is treated differently from other kinds of data. This leaves it in the paradoxical position of being both a privileged type, in that it is the document and everything else is a transclusion, and a second-class type, in that it is sort of walled off from the rest and cannot be manipulated independently of the inlined meta-data.

    • The standard links are only usable as clickable hyperlink markers, which not only is very limited (compared to the more general meta-data approach Xanadu folks favor), but also binds the link to the structural and presentation layouts of the document.

    • Formatting is also inline, making it hard to separate data from structure, and structure from presentation. Even if you were able to transclude data freely, you'd have to either live with the formatting it already has or use a script to strip the formatting out and insert your own.

    • HTML tags for specific types of objects are part of the language, and adding a new category of objects to support requires changing the language itself, either by bolting on a proprietary extension or by making a series of proposals and debating them with the standards committee. Conversely, support for a specific format is in the hands of the browser developers, and those developers have no obligation to support new tags beyond improving their market share. This means that most new things end up supported haphazardly, through browser extensions or client-side scripting.

    • URIs refer to filenames rather than to arbitrary data, and the only standardized way to kink to a part of a page is to have an anchor-target tag in to document itself. Since the links also have a starting point but no ending point, you cannot transclude just the part you want to without scripting.

    • URIs point to a specific server name and file name, and can go stale if the domain name or the filename changes, or the server becomes unavailable.

    And now, to counter my rather fanboy-ish enthusiasm for the moment, I should mention that most of these points were actually part of the reason why WWW succeeded and Xanadu failed. The simplicity of the original SGML-derived HTML was such that anyone could write a text document, add a few basic tags, and have a web page. Since this was all Tim Berners-Lee was looking for at the time, this worked great. Similarly, since the HTTP and HTML specs were so simple, and available to anyone, even a half-assed coder could kludge up an HTTP server or a simple web browser. This make the next step of the development, Marc Andreesen's <img> tag proposal, a simple step to take, and set the whole popularity of the WWW in motion. Meanwhile, between excessive ambition, chronic CADT, terminal Babbage's Syndrome, and the endless loss of funding sources when the schedule inevitably slipped, Xanadu never progressed before or after the WWW came along.

    It is sort of the definitive case of 'Worse is Better', because in this case 'worse' was workable if kludgy but 'better' may never even be finished.


  • area_pol

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    problems fans of Xanadu have with HTML and HTTP in general

    It is sort of the definitive case of 'Worse is Better'

    Leaving the history aside, HTTP and Xanadu have a different use case.
    Xanadu means serving static content.
    HTTP is for using a remote application on a different machine, with HTML being a way of serializing the UI (quite like SSH's X-server tunneling). This application may be a static file server, but may also be email client, shop, game or file editor.
    URIs do not refer to files, but to inputs to that application.

    The HTML you see on usual sites has long lost any semantic meaning, and is rather about precise rendering instructions - similar to what you would expect if you looked inside a PDF.

    So there is no point in putting HTTP and Xanadu in opposition.
    Xanadu could never replace the dynamic-remote-application use case of HTTP.
    But it could very well be one of the applications that display their UI through HTTP, because that makes it easy to use by many people.



  • @Adynathos said in ...did Kubla Khan a stately pleasure-dome decree...:

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:
    Xanadu means serving static content.

    0_1462196982516_Shive_What_small_45.jpg

    To paraphrase Charles Babbage, I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a claim.

    Seriously, that's like arguing that an NFS file system is mean for serving static content. It's so fucking wrong that words cannot describe it.


  • Discourse touched me in a no-no place

    @ScholRLEA said in ...did Kubla Khan a stately pleasure-dome decree...:

    URIs refer to filenames rather than to arbitrary data,
    [...]
    URIs point to a specific server name and file name,

    Nope. :)

    URIs identify a resource. (The clue is in the name: Uniform Resource Identifier.) That resource might be implemented by a file, or might not. That's part of the implementation of the resource, and not something that either the referent or the consumer of the resource ought to be concerned about.

    Exactly how a URI identifies a resource varies quite a bit; the scheme part tells you. For HTTP (and its variant, HTTPS, as well as a few others such as FTP) a server is indeed identified; that's how that scheme works. The simplest implementations will map the resource to files. There's a lot of more complex implementations (e.g., you need to study RESTful webservices; they completely ignore your preconceptions about mapping to files). But there are other schemes that are radically different.

    • doi: This scheme is a lot more common in scholarly works than anywhere else now, and uses the concepts of “well known resolvers” (e.g., http­://dx.doi.org/) to allow references to resources that will persist for decades. Fixed resolvers? Pah!
    • data: This scheme embeds the referred to data directly within the document. It's relatively common in CSS (which is also a part of how other things in your criticism don't really apply any more).
    • magnet: This scheme is all about obtaining the resource from a distributed P2P system like bittorrent. The thing you describe in the identifier is a hash of the content you want; where it comes from is entirely out of band.
    • urn: The URN scheme is pure name. It's not necessarily resolvable at all, and is ideal in situations where it is the identifier itself that matters, not what it resolves to.

    I'm sure there are others.

    Also, resources are not necessarily single fixed binary representations. They can be a family of representations (e.g., with the right one chosen by content negotiation) and they can also be entirely dynamic.



  • @dkf Sorry, perhaps I should have said URLs then. I was specifically speaking of HTTP page links (as they were originally defined by TBL, at least), not general URIs. That was careless of me, I'll admit, and I should have known better because I have in fact used other types of URIs.


Log in to reply