Monday, October 19, 2015

What's taking you so long?

WildStar Reloaded is here since almost three weeks, and you might be wondering what is taking me so long to update Jabbithole to the new patch. So here it is, a little behind the scenes info for the curious.

First and foremost, I had no beta access at all, so I could not prepare for the upcoming changes ahead. But even if I had I could not do too much with it, as the beta schedule and the launch was overlapping with deadlines of two other projects I've been working on (with one of them still in progress, taking up most of my time). Long story short, I started working on the update at launch, with a somewhat limited time available.

Rescuing a kitten along the way certainly did not help either. Caring for a six weeks old furball requires a lot of attention, therefore time. It's well worth it though, he's a marvelous little rascal.

Jabbithole itself is operating on four main, very different and distinct levels. There is static data pulled directly from the game files, there is dynamic data collected by the addon, there is the import process that puts these together in a consistent way and there is the website itself, presenting the data to the players like yourself.

In order to be up-to-date, all four components has to be in sync: I have to be able to read the game files, I have to be able to collect data with a current addon, I have to be able to process this data, and the website has to handle this data properly.

Patches, especially major patches took me usually a week or so to bring all of the above up to speed (and most of the time I was lucky with spare time too), but WildStar Reloaded introduced way more changes than usual.

In Reloaded Carbine changed the compression format used for the internal game files, so my first encounter with the patch was like this: Holy crap, I can't read a single file. Now, Jabbithole only utilizes a small portion of the game files, but being able to access these is rather vital. It sparked the quest to ogle hexadecimal byte sequences and match these against known compression formats, whilst praying that Carbine didn't decide to use a proprietary format this time. Luckily they did not, so in a couple of days I was able to read said files again. Then came the surprise, all of the textures were garbled. By the look of them it seemed to be an offset issue, so I started looking for extra headers and the amount of extra padding I need. Once I've found it the majority of the textures were nice again, but some of them were not. It seems that the packed texture format has been changed in some cases, and since juggling with 3d texture formats is definitely not in my wheelhouse, I am looking at a good few more hours before I can crack those as well. Luckily, these are cosmetic issues only, the worst case scenario is that Jabbithole will run with old/placeholder icons and maps for a while. Beware Carbine, I might be coming your way begging for some hints in the foreseeable future.

Once I was sure that things are looking fine regarding the tasks the farthest away from my area of expertise, I was able to start working on what I am familiar with: the data collector addon. It did not came as a surprise that the patch broke almost everything, patches are like wild animals: they look nice, they look cute, and when you get your hands on them they bite it off in an instant. Carbine developers love to rename (tMaterials vs arMaterials, I am looking at you) and spice up things between patches, so it's time to grab the shovel named Rover and start digging deep.

Ironing out the most obvious kinks was easy, the client throws Lua errors when a method name no longer exists. Most of the time it is a trivial task, but there are some gems, like when a complete subsystem gets revamped (challenge rewards and item stats, for instance) or when functionality is moved around into new packages (housing stuff). But it is relatively straightforward to fix these issues, you only need to fiddle with Rover and check out a few Carbine addons to get what you are looking for.

But usually there are more stealthy changes as well. When a property name no longer exists Lua gives you a nil value instead of an error. So when someone at Carbine decides that it will be better for the next couple of months to call the list of materials needed by a schematic arMaterials instead of tMaterials, the only thing you'll get is no list of materials whatsoever for any of the schematics. This means one thing: you have to match each and every data saving function of yours against the dump you save and look for missing stuff.

In Jabbithole's case, that's about fifty functions spread over a good 3500 lines, with most of them being a data assignment instruction using one of those pesky properties whose current names for this week is only known by a blind hermit locked away in the basement of the Carbine HQ. So what you do is you generate as much data as you can, and you go over the code line by line and check if that property is present in the save file or not. If so, you smile, if not, you fire up Rover and start digging. Again.

Of course this is only true for not-so-obscure data, but there are plenty of things that require a bit more work on your part. Veteran NPCs? Go to Galeras, enter Stormtalon's Lair, launch Rover, and guess what: start digging. Missing rune budget values? Time for some runeslotting. And some Rover. Invisible Loot Units? Go to an adventure and work through it until some reward drops. Did I mention Rover? And the list goes on.

As you can guess, this is a rather tedious and slow process (albeit game time well spent on gazing into the soul of WildStar), and one that can not be done in a half-arsed way, unless I want to publicly apologize a few weeks later: Sorry gals and guys, your collecting efforts have been in vain, all the data you've sent in is inconsistent, for I was a lazy bastard an did not properly check everything, so let's start over! I say let us not.

Hiccups might occur of course, but tonight I'll be sleeping well, the Jabbithole addon version 0.2.0 released today is as thoroughly triple-tested, as humanly possible. So please fire it up, and start sending data I can be crunching on.

This is where we are now, the data importer still needs to be updated, then the website has to be brought in sync with the database, as plenty of data structures have to be changed in the process. Then finally, you'll be looking at a shiny new Jabbithole with a plethora of up-to-date information.

Originally I was hoping to finish everything over the next week, but it slipped my mind that I'll be out of the country for a few days (Ireland, beware!), so realistically it won't be up and running before the end of October (one has to work as well to feed them cats). The silver lining is that in the meantime you can send in a lot of new data. How fun is that, right? ;)

Thank you for your support and your patience, and enjoy Jabbithole, past, present and future alike.




  1. You have our support (and data)! :) Keep up the good work.

  2. I am very eagerly awaiting the next update... I feel so handicapped without this database! ;)

  3. I feel bad asking, but any update on the progress mate? (:
    Really longing for this !

    And thank you so much for keeping the site up and running mate

  4. It's been over two months, and I know I religiously send data from my addon nightly. I also feel bad, but is there any ETA?