More Projects

The state of JSONView in 2017

I released JSONView in early 2009 to solve a simple problem I had - I’d been working with a number of JSON APIs, but Firefox had no way of directly viewing JSON documents. At the time, it wouldn’t even display them as text, showing a download dialog instead. Even after Firefox started showing JSON as text, it was still useful to show JSON nicely formatted and highlighted rather than however it was served. In the nearly eight years since its launch, JSONView has grown to about 130,000 daily users and has been downloaded 1.5 million times.

Once the initial few versions were out, I’ve updated JSONView on average every six months. Following that trend, I released JSONView 1.2.0 in late November. I’ve expected Firefox to subsume JSONView’s functionality ever since I first launched it, but amazingly this hasn’t happened yet. That said, Firefox has recently gained a very nice native JSON viewer hidden behind a developer flag (that’s only on by default in Firefox Developer Edition) and JSONView now offers the option of simply turning on that viewer instead of doing its own thing. Hopefully soon, JSONView won’t be needed at all.

I’ve always applied a minimalist philosophy to the extension. I’ve resisted adding a lot of bells and whistles in favor of a simple extension that does one thing well. This has the side-effect of reducing the rate of change of the extension, since besides the constant work of keeping it up to date with changes to Firefox, it’s pretty much done. There are still capabilities I’d like to add to JSONView, but those that haven’t already happened are usually impossible within the bounds of Firefox’s extension API. Speaking of which, Firefox is generally moving away from the sort of deep integration for extensions that JSONView requires - I hope JSONView is completely obviated by the time it can no longer work at all.

I personally don’t use JSONView anymore at all, because I no longer use Firefox. When I interact with JSON APIs I make use of the Chrome devtools, Emacs restclient-mode, and the jq and gron command-line tools. I’ve continued to maintain the extension, which remains open sourced on GitHub with several contributors helping out over time.

JSONView has been cloned, without permission (but without any resistance - it’s open source after all) as a Chrome extension in at least two instances. One of them became very popular but has been abandoned and is no longer on the Chrome web store because it contains security vulnerabilities. I never made a Chrome version myself because Chrome doesn’t offer the same integration APIs that Firefox does, so I couldn’t make it work the same way. JSONView for Firefox intercepts requests for JSON documents and produces HTML instead. The Chrome ports run on every single page that Chrome loads and try to figure out whether it looks like a JSON document, then reformats the document in-place. I never liked the security and performance implications of that, so I’ve avoided an official port. I keep an eye on Chrome’s extension API in case it ever becomes possible to do things the way I want to.

In summary, JSONView the Firefox extension is doing well, but is a mature project that will change slowly, if at all. JSONView for Chrome appears to be dead, and I don’t have immediate plans to produce an official port myself, but I won’t rule it out and I’d accept a well-implemented contribution as long as it matched the design and implementation philosophy of the original.

Save time runnning FindBugs by excluding SQL checks

FindBugs is a great tool to help you find tricky problems with your Java code, as well as to learn more about Java best practices. Unfortunately, for a large codebase, FindBugs can be extremely slow, often taking several minutes to run. In the last few years, the Java projects I’ve worked on haven’t made use of SQL databases, and I recently discovered that FindBugs’ checks against SQL injection make up a major portion of the overall runtime for the tool. Try creating a findbugs-exclude.xml file, and putting the following in it:

<?xml version="1.0" encoding="UTF-8"?>
<FindBugsFilter>
  <!-- We don't use SQL or JDBC, so remove these very expensive bug detectors. -->
  <Match>
    <Bug code="SQL,ODR" />
  </Match>
  <Match>
    <Bug pattern="DMI_CONSTANT_DB_PASSWORD,DMI_EMPTY_DB_PASSWORD" />
  </Match>
</FindBugsFilter>

In my experience, this shaved a whole minute off a multi-minute FindBugs run, with no downside since we don’t use any SQL or JDBC. Of course, if you do use those technologies, these scans are probably still worth the time!

Node.js stack traces in Emacs compilation-mode

, ,

I’ve been using Emacs regularly for at least 10 years, for all of my programming tasks that don’t involve Java or .NET. Amazingly, for most of that time I’ve used a pretty stock installation of Emacs with only a few cargo-culted customizations stolen from my co-workers’ .emacs files. However, as part of switching to a new job in the last month, I’ve taken the opportunity to invest in my shell and editor, and I’ve fallen deep down the rabbit hole of Emacs customizations. My continually-evolving config is publically available on GitHub. The difference between stock Emacs and one loaded up with custom packages is astounding - I can’t believe that I had been missing out on all of this. But this is not an article about setting up Emacs to be your perfect editor. Instead, I wanted to provide a tip for Node developers that I hadn’t found the answer to when I went looking.

The other day I was working on a Node.js script, and I was constantly switching back and forth from Emacs to the terminal to run the script, hit some error, remember the line number, go back to Emacs to look up the error, etc. Then I remembered that Emacs has the compile command, which will run a shell command and show the output in a separate window. Now I could split my frame into two side-by-side windows, edit my script on the left, and run compile to test out my script with any test command I liked (and then use recompile or g in the *compilation* buffer to re-run the command). But there’s another neat feature of compilation-mode - it can understand stack traces and provide hotkey access to the line that triggered an error. Unfortunately, the stack traces from Node were not being recognized correctly, and Emacs thought the last number was the line number, when it’s really the column number.

To fix this, I simply needed to add a new regexp to the list of patterns compilation-mode understands:

;; Add NodeJS error format
(setq compilation-error-regexp-alist-alist
      (cons '(node "^[  ]+at \\(?:[^\(\n]+ \(\\)?\\([a-zA-Z\.0-9_/-]+\\):\\([0-9]+\\):\\([0-9]+\\)\)?$"
                         1 ;; file
                         2 ;; line
                         3 ;; column
                         )
            compilation-error-regexp-alist-alist))
(setq compilation-error-regexp-alist
      (cons 'node compilation-error-regexp-alist))

Writing that regexp would have been hard, but of course Emacs has a solution. First I found the current value of compilation-error-regexp-alist-alist using describe-variable in order to get an example regexp to start with. Then, in the compilation buffer where my stack trace was, I ran re-builder, which pops up a little mini window where I could tweak my regexp and see immediately whether or not it was matching correctly. Then I just added this to my init.el, evaluated it with eval-region, and my next compile had the errors highlighted correctly, and I could immediately go to the offending code with next-error or by putting the point on the stack trace line in the compilation buffer and pressing RET.

Hopefully this helps out some fellow Emacs-using Node developers, and serves as a reminder that your tools can always work better for you.

Converted GoPro Videos Don't Work in iMovie on Yosemite

,

Update June 25, 2015: iMovie 10.0.9 has been released which fixes the problem I describe below.

Recently, I’ve been enjoying using my GoPro camera with iMovie to put together silly little videos. I recently upgraded my MacBook to OS X 10.10 Yosemite, and at the same time accepted the automatic upgrade of iMovie to version 10.0.0.6. After doing so, I realized that all my previously-imported GoPro videos wouldn’t play in iMovie anymore, showing as “empty” tracks instead. I also couldn’t import converted video into iMovie anymore.

iMovie not playing GoPro Videos

To back up a bit, I’ll explain how I usually work with GoPro videos. I shoot video with my GoPro using the “ProTune” mode, and then import those videos onto my Mac. From there, I convert them into an editing-friendly CFHD (CineForm HD) format using GoPro Studio. GoPro Studio understands ProTune, and offers nice color correction and exposure controls - much better than iMovie can offer. After color correcting the clips, I’ll import them into iMovie and then deduplicate them using a script so that I can still use GoPro Studio to make further color tweaks while editing in iMovie. I can then use iMovie, which I quite like, to edit my videos, add music, etc.

Unfortunately, that workflow is now broken because iMovie no longer recognizes the CFHD video format. It can import the original, pre-converted videos from the GoPro, but thanks to the ProTune recording mode, those videos require extensive color manipulation to look good (which GoPro studio does as part of its conversion process). To work around this limitation, I’ll either have to switch to doing all my video editing in GoPro Studio (which is pretty limited) or export my videos to an h.264 encoding from GoPro Studio so iMovie can use them. I’m not really excited about the latter option as that means yet a third copy of my video, and I won’t be able to tweak videos in GoPro Studio after I’ve exported them.

I’m not sure exactly what’s going on, but if I were to hazard a guess it’s that OS X Yosemite and iMovie 10.0.0.6 are continuing with Apple’s migration away from Quicktime (which allowed for third-party video codec plugins, like CineForm’s CFHD plugin) to AV Foundation, the media framework from iOS that Apple has brought back across to OS X. Unlike Quicktime, AV Foundation only supports a limited set of video formats, and does not allow for external plugins that add support for other formats. This is why, for example, Perian no longer works. I had hoped that iMovie, being cut from the same codebase as Final Cut Pro, would be exempt from this dumbing-down. I wonder why this happened, without announcement, and in a point release of the software. I’m also not sure, if this is really the case, what GoPro/Cineform could do about it now that Apple has closed off any avenue for third parties to extend OS X’s media capabilities. It certainly doesn’t make me want to jump up to Final Cut Pro and wonder whether things would be better supported there.

That said, I haven’t done a lot of extra research to validate my theory. I haven’t confirmed that other formats no longer work with iMovie, and I haven’t tried a Mac with Yosemite but without the new iMovie to help narrow down which one may have introducted the change.

I’ve searched around for reports of other people having this problem, and I’ve found a few Apple support forum posts (no answer) but not much else. I opened a support ticket with GoPro, but they were clueless and eventually stopped responding to me. My hope is that by documenting this problem, this post will at least be somewhere for others in my situation to commiserate, or even better, to tell me I’m wrong and I just need to do X to fix everything.

Hard linking videos to save space in your iMovie library

,

After an initial adjustment period, I’ve come to quite enjoy the new iMovie (2013). While it initially appears to be oversimplified, it actually still has most of the editing and organizational features that were available in older versions of iMovie, and it moves to a Final Cut Pro X based backend that is much faster and more powerful. In fact, it’s pretty clear that the new iMovie is just a reskinned, limited version of Final Cut Pro X. The best part about this is that many operations that used to block and make you wait in older iMovies are now done asynchronously, allowing you to keep working while clips import, movies render, or stabilization is analyzed. This is a huge win even for the small, simple videos that I put together of scuba diving or flying my quadcopter.

One thing about the new iMovie that I strongly dislike is that it now consolidates all your files into an opaque “library” which can only be used by a single application. Any videos that you import into iMovie get copied into this monolithic library, and the individual files cannot be accessed from any other application. This means that you can’t use other editing tools on your video – for example, I prefer use the free GoPro Studio app to do color correction for my GoPro videos, but I can’t open the copies inside the iMovie library. iPhoto works in a similar way, wher all your photos get placed inside a library and are no longer accessible to other tools like Pixelmator. I find this frustratingly restrictive. I prefer to use the filesystem to organize my files, and to be able to use multiple tools in my workflow.

The other problem with the library concept is that iMovie copies videos into its library, meaning you now have two copies on your disk. I have a 512GB SSD on my MacBook, so I don’t have the space to leave duplicate files everywhere. I suppose Apple intends for you to only keep the copy in your iMovie library, but like I said, I prefer to have my files exposed on the filesystem so I can edit them or organize them individually.

To work around these problems, I wrote a very simple Ruby script called dedup-imovie-library. It’s available on GitHub, and you can just drop it into some location on your PATH. The script simply searches your iMovie library (which is really a folder that the Finder treats specially) for movie clips, and finds matching files in a second folder that you specify (the original source of those clips). When it finds a match, it deletes the copy in your iMovie library and replaces it with a hard link to the original.

For example, if my library is called iMovie Library.imovielibrary and my originals are in a folder called originals, I can just open Terminal and run:

cd ~/Movies
dedup-imovie-library iMovie\ Library.imovielibrary originals

Now, those files will exist in two locations but only take up space on disk once, because they’re hard linked to the same file. As an additional benefit, any editing I do to the originals (like color correction) shows up in iMovie, because they’re both using the same file.

There are a few caveats:

  • You still have to import your files into iMovie using the Import command, and wait for them to be fully copied before you run the script.
  • You should close iMovie before using the script, to avoid issues when the clip is replaced.
  • You can only do this when your originals are on the same disk as your iMovie library - hard links can’t go across disks.
  • Backup software like CrashPlan and Time Machine don’t really understand hard links, and will still back up two copies of your files.

Another option besides hardlinking the files together is to use a symbolic link instead. Symbolic links are simply a pointer to the original file, and as far as I can tell iMovie 2013 has no problem with using them. Symbolic links would solve the problems of not being able to link across disks, and would not take up double space in backups. However, iMovie ‘11 had a lot of weird behavior when using symbolically linked files (such as not being able to “favorite” sections of a clip) and I haven’t had a chance to test this with iMovie 2013. I’m also concerned about what would happen if I dragged an event into another library where the videos were stored as symbolic links. I’ll try out the alternative at some point and write another article if it works well.