I need a simple, light database to accumulate some data from a JSON feed. Ideas?



  • I have a little personal project I'm working on. I know what I want to do, but I'm not really sure how to accomplish it.

    I have a JSON feed that hands out song names and artists, and I'd like to compile them into a database so I can keep track of 1. all songs that were played and 2. how often each song was played in a particular time period. My experience with databases, however, boils down to installing some MySQL-based web apps and using Access in high school.

    Are there any nice, light programs I can use? Perhaps a scripting language like Python or Ruby (I'd have to teach myself) with some sort of database plug-in? I've heard of SQLite and the concept sounds about right, but I don't know SQL and I don't know how I'd interface between the database and the JSON data.

    I was attempting to use Yahoo Pipes until I realized the whole point of Pipes is to [b]create[/b] a JSON-like feed of data, which I already have.



  • CouchDB comes to mind. It's a NoSQL Document DB like MongoDB, but especially designed for JSON-structured data and accessible via HTTP. (Apparently there is also a free cloud service for personal projects) It should be sufficient for your use case though it has some catches for more sophisticated data models. (read: anything that requires a join)

    Or, of course, you could just learn SQL. For your project, you should be fine with some basic SELECT and UPDATE queries and it'll probably be worth the investment in the future.



  • I've played a little bit with MongoDB, which is ideal for this purpose because it "speaks" JSON for everything. (Not just data feeds, but sprocs and such.)

    Learning curve is steep though.



  • Hi,

    maybe it's not so widespread but Ojota (http://ojota.readthedocs.org/en/latest/) it's a nice json based database that works either with the filesystem or a feed coming from the net. The code is also easy to understand.



  • @evilspoons said:

    I have a little personal project I'm working on. I know what I want to do, but I'm not really sure how to accomplish it.

    I have a JSON feed that hands out song names and artists, and I'd like to compile them into a database so I can keep track of 1. all songs that were played and 2. how often each song was played in a particular time period. My experience with databases, however, boils down to installing some MySQL-based web apps and using Access in high school.

    Are there any nice, light programs I can use? Perhaps a scripting language like Python or Ruby (I'd have to teach myself) with some sort of database plug-in? I've heard of SQLite and the concept sounds about right, but I don't know SQL and I don't know how I'd interface between the database and the JSON data.

    I was attempting to use Yahoo Pipes until I realized the whole point of Pipes is to create a JSON-like feed of data, which I already have.

    Unless you're wiling to learn not just SQL but systems programming as well, I'd stay away from Sqlite. It's a great project in many ways, but it's not very noob-friendly. Personally, I gravitate toward Postgresql for most new development. The SQL you'll have to write ought to be pretty minimal.



  • Well, I decided to axe the 'simple, light' requirement and I just went ahead and installed a full LAMP stack in a virtual machine. I've started diving in with MySQL, and I must say that it's not quite as hard as I thought it was.

    I've managed to achieve my goal of downloading the JSON feed and inserting the data into a couple tables, and I'm starting to learn how to manipulate the data in such a way that I can get the results out of it that I was interested in finding.

    I think the biggest advantage of changing from "something light" to a full blown Apache/MySQL/PHP installation is just the quantity of documentation and tutorials available for everything.

    Thanks for the suggestions anyway, though!


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.