Kindle RSS Ripper

After trying out blogs and periodicals on the Kindle, I'm left reasonably impressed. It's really neat to pick up the Kindle and find that one of the blogs you watch has an update you haven't read yet. The push functionality is cool, and I dig what amounts to the pay as you go approach to the EVDO bandwidth. I mean, someone has to pay for it somewhere, right? And so what if Amazon are playing the margins for profit, the ARE a business, right?

But what about RSS feeds that aren't on the Kindle service yet? For that matter what about RSS feeds that you don't care about quite so much as to be that up to date? I have a few blogs that I read once in a blue moon, and entertained as I am by them, I can't see shelling out cash for them. Clearly Amazon don't want to trumpet free content when they're offering their upgraded cool stuff for pay, but they've graciously left a trail of breadcrumbs for anyone so inclined to follow.

First off, the format of Kindle documents appears to just be a wrapper around Mobipocket files. These are compiled binary files created by tools available at the Mobipocket site. Believe it or not, there are free versions of the tools there, including a command line compiler.

What are the input files? Turns out they're just plain old HTML. Mobipocket recognize a few extra tags aimed at books (e.g. a page break tag), but otherwise they're just HTML. Reference for the whole thing is available at the site.

So we have XML (RSS) feed documents on the one hand, and a command line compiler that takes HTML files and builds our desired output. Piece of cake!

I've whipped up a little command line utility of my own in C# that will process a list of rss feeds into a set of .mobi Mobipocket books. I'm posting it here for anyone else to try, but I make no guarantees about whether it'll work for you, nor will I offer any support or additional features. I don't have a lot of free time these days, and I'm only likely to work on it further should I need something more of it. Hey, if we're lucky this will inspire someone else to go make a fancier version!

Having said that, this does work reasonably well. So far I've tried it with Kotaku, Slashdot, GayGamer, XKCD.com and a bunch of other sites. If it fails for you on some other feed, I'd be inclined to suspect that it's yet another weird variant of what it means to be an RSS feed. I didn't know that the whole concept was such a mess!

To run this:

• Download and unzip the RSSKindleSync.zip from here file to any directory of your choice.
  
• Download the mobigen utility from the Mobipocket site and copy the actual mobigen.exe file to the same directory where you left RSSKindleSync. You should now have 4 files there: RSSKindleSync.exe, RSSKindleSync.exe.config, feed list.xml and mobigen.exe.
  
• Now edit the feed list.xml file in that folder to add all the sites you're interested in ripping. Just copy and paste the existing tag and change the URL parameter for each.
  
• Run the RSSKindleSync.exe file and away you go!
  
• You should end up with a pile of .mobi files in a /books/ subdirectory under that exe file, one for each RSS feed you listed. Just copy these over to your Kindle's documents directory and you're done.
• Whenever you'd like a refresh, just rerun RSSKindleSync.exe and all your RSS books will be rebuilt.
  

Enjoy!

Amazon's Kindle, Device of Joy

So I've been waiting a decade or so for this thing. I grew up in an area far from easy access to books, and I was a voracious reader. After one point, appreciated presents almost entirely boiled down to new reading matter. Sure toys were great, but books were awesome. In the afterglow that was finishing a good book, I regularly fantasized about a magical book that I could command to show me any book I could think up. In that pre internet era, it was a pain just looking up the rest of an author's work, leave alone actually getting my hands on it.

So here now is the Kindle: an absurdly light, comfortable, easy to use little panel that can call up any one of 90000 books and growing. Sure, it's powered by money instead of magic, and it's surprisingly hideous to first glance, but it works. Damnit, it WORKS.

I'm as skeptical as the next guy, but the pent up lust I have for what this device promises was so strong that I put cash down first thing Monday morning. I practically ripped the packaging apart to get to it, and I'm happy to say that it's a bloody great toy.

The screen is wonderful, and a little magical. It is very comfortable to read off of, and the bizzarely physical nature of its rendering is a fun novelty. It's weird to think of it actually physically moving little elements around to form the picture. It's still all very first gen feeling: the refresh rate is awful, and the colors (think dark graphite pencil on newspaper stock) are clearly the only thing the technology supports, rather than a conscious design choice. But it WORKS.

Speaking of design, the thing is absolutely as hideous as the pictures make it out to be. It looks like a 10 minute quickie prop from a half discarded Buck Rogers episode. There's no defending the thing's looks, from its abundance of gawky angles to its flair-fallen-flat kooky keyboard. Gods willing, Amazon are already on a complete redesign of the form of the thing. But again: it works. The UI is limited by the screen's refresh rate, but they've done a great job compensating for that with the cool mercury slider bar. The giant page forward and backwards buttons are perfectly positioned for me, and the keyboard is quite serviceable for the uses its put to. It's also a lot smaller than the pictures make it out to be.

Book vs Kindle then.

• Instant on: I can pick up a book at any point and thumb right over to where I left off. I routinely just walk away from a book to answer the phone, or the door, or to greet friends walking by, and so on. I've tried reading on my tablet PC, which fails miserably here. I'm always aware of battery life, and am very much in the habit of remembering to put it to sleep if I think I'm going to be away from it for a time, leading to a wake up wait the next time I want to pick it up. The Kindle on the other hand, I have no such qualms about. I've been using it for days without charging it up (keeping the radio off most of the time), and walking away from it is a non issue: it takes no power unless it's actually rendering a new page. Verdict: Tie.
• Easy on the eyes: Books are a reflective tech. They rely on ambient light and therefore naturally match the contrast of the rest of your environment, meaning that your eyes don't have to adjust much to look at it. I certainly find far less strain from reading a book than from working on a PC for hours. Laptops, Tablets, iPhones, etc, all suffer the same sort of problem. The Kindle's screen is also reflective, and so as comfortable as a book. As a bonus, when my eyes are feeling a little tired, I can zoom in on the text and read with a larger font for a while. Verdict: Kindle wins by a slim margin.
• Reading Comfort: Books are pretty small, light things that are easy to hold in one hand and therefore lend themselves to being used in all manner of comfortable positions. Curled up on the couch, lying in bed, lounging by the pool. My Tablet PC can do some of this, but it's just plain too bulky and heavy to lend its self to the one handed thing. The Kindle is as light or lighter than your average paperback, and about the same size. It's more than easy to hold while reading and has the added bonus of being always flat unlike the sometimes unruly curling pages of a book. No more annoying shadow creeping up from the spine if you fail to hold the book open far enough, no wind blowing pages over, no annoying imbalance in weight when you're at the head or the tail of a book. It's also the same size and weight no matter what size the book. No more struggling with that giant space opera epic at the breakfast table! Verdict: Kindle totally wins.
• Price: The Kindle is WAY too expensive. I bought one, but I'm a nerd. If this is going to sell to everyone in the Oprah book club, it needs to be more like $100. Heck, if it was $50 and sold at airports, Amazon wouldn't be able to manufacture enough to keep shelves stocked for years to come. The actual e-books themselves are decently priced; I've bought a half dozen so far and none have been more than $6. That's absolutely fine with me. But the giant outlay here means books win this battle by a comfortable mile.
  
• Convenience: I've moved around a lot in my life. The combination of being a nutcase reader and a nomad has meant that I've abandoned stacks upon stacks of books to charities, old boyfriends and (in the case of particularly dire books) the local landfill. On occasion I've almost wept at the separation; there was this one particular move after the period where I'd discovered Heinlein where I had to get rid of his entire body of work. Just plain ouch. Sure Amazon won't be here forever, and there's some chance that my collection will get bricked at some point, but they've been in business at least 3 times longer than the longest I've managed to hold onto all but a literal handful of books. Kindle pwns this one.
• Accessibility. Books have become a hell of a lot easier to get at then when I was a kid. Even rare books are relatively easy to find online. The Kindle still has a limited supply of titles, but with Amazon behind it I have faith that more will pour in. I mean, if Amazon can't do it, no one can. Books that are available are absurdly easy to get at. The built in store is a thing of beauty, and already I've ended one book only to desperately bash at the keys to get my next fix from the same author (Neil Gaiman in this case. I hadn't read Anansi Boys yet, and segued into previously unknown to me Coraline). Less than 5 minutes I think it was, and I was sitting in a cafe to boot. I think 8 year old me wants the now me dead. Kindle for the win.

So all in all I'm happy with this evolution in reading. After thinking about it for a while, I find it silly to compare this thing to a book. It IS a book. We're not used to seeing advances in book technology, but this is it. It has plenty of room to grow in every direction, but the device that's available here and now works as advertised and the service it fosters will be the blood that runs the next generation of books.

Ugly as it is, it's certainly not leaving my side for months to come.

Doh!

To everyone who's e-mailed me in the past months since that last post: sorry I didn't get back to you! Life's been stupid busy, and what with the move to the US, I was cut off from any serious internet access for ages.

To those interested in what's become of RPG Zero, it's still sort of alive back there. I'm afraid that despite going for a job that I thought would be more relaxing than the last one (which was an overtime filled nightmare), I'm still finding myself without enough free time to work on personal projects. C'est la vie, eh? I still want to make RPG Zero, though now it's more likely that I'll be doing so in the quiet holiday/vacation moments rather than trying to jam it into evenings and weekends. I'll post here when and if I make any major strides.

Thanks again for all the kind comments and e-mails. And good luck to everyone still working on their projects!

Pulp Friction

Karl in a comment asked what I'm doing for collision, so here's a brief run down.

I export the levels from Max as a simple XML file with triangles, load that and parse them into a friendly structure. I store all the parsed triangles in a balanced binary tree, subdividing cells on longest axis. I would use an octree, but that requires subdiving for effect, and I haven't yet invested in writing a triangle splitter.

My movable physical objects are all spheres.

After I move any of the physical entities, I do a constraint pass against the level, i.e. for each triangle, check to see if my sphere is intersecting with it. If it is, move me out of the triangle, using the direction I'm coming from, i.e. the direction from where I currently am, to where the intersection happened. This works just fine as long as none of the entities can move farther than their diameter in a single frame.

I do the same thing with any other dynamic physical barriers, but they are represented by boxes.

I also provide ray-tracing onto the level mesh, which allows for queries like, "find the closest point on the level beneath me".

Entity to entity collision is straightforward sphere to sphere intersection tests. Again, works just fine if no one can travel faster than their diameter in a single frame.

To avoid noise, separation between entities isn't direct collision response as it is with the level, rather it is applied as a separating force over time.
In effect, the wolves are flocking away from each other rather than strictly colliding. The thinking here is that it's better to have two wolves intersect briefly rather than bounce back and forth as they're corrected violently each frame. This also means that by and large they can't get each other stuck.

Let There Be Cowbell

As promised, here's a look at a few pretties. We have a level segment with a new palette, and a new badguy, the froggy. These are the entry level badguys, so they don't put up much of a fight. Still blow up nice though.

I've also snuck in the healing item in the game: mushrooms.

So, coming on from last time's discussion of mesh animation, this update shows more examples of it. All the effects and character animations are done in the mesh animation manner, and hopefully to a consistent style. I still like it, still planning on writing a more efficient Content Pipeline plugin to pack them for me.

The art style is still holding together, I think. There's enough range in it, that I think I can cover all the locales I'm thinking of, and the gradient in the character of the environments I hope to depict over the story. I wish I had some time to sit down and concept out, but honestly at this point it's more profitable to just plow along. It takes an hour or so to model a new badguy. Bouncing back and forth between code and art (ignoring the vertigo), it was about 5 hours to get the frogs up and running from scratch. It's still a discovery process though, so I'm not too hung up on polishing yet. Getting everything working is first priority.

I have the design for the first level laid out now. It happens entirely underground, in a Zelda like dungeon configuration, complete with boss character. I'll stay focused on getting that done before moving on with any other portion of the game. I'm considering putting it up for public download at that point, to get a little feedback on how it feels.

Depends, really, on how embarrassing the whole thing is by then.

Before anyone asks, there's nothing clever about the cave scene. Reflections are the good old fashioned "render everything upside down", and the caustics are nothing more than a bit of fancy texture scrolling, where the x component is a sin wave on the y component. Sometimes, you just don't need clever to be fancy.

I'll be moving home for the next few weeks, so this might be my last post for a while. On the bright side: I'll be getting some new gear when I get to my new place, so perhaps I'll get around to trying some of those clever bits. At present, this PC is pretty much nipping at the limits of its little NVidia Ti4400 graphics card!