When it comes to frontend creative software for multimedia projects, Blender and Audacity tend to get the most attention, mainly because they are easy for beginners to use. But when you use them to produce a large multimedia project, like our animated open movie project Lunatics!, you quickly run into problems with the backend infrastructure.
You might not even realize at first that your problem is rooted in disorganization. You find yourself procrastinating and/or working in circles because you can't keep your project organized in your head. You need software to help you.
One of our goals has been to solve this problem using open source software, and thus remove obstacles to doing longer-format movies on the non-existent budget of a free-culture film project.
Digital asset management
The first and most obvious problem is called digital asset management (DAM), and a system to solve that problem is called a "digital asset management system" (DAMS). An "asset" is a component of a multimedia project, be it an image, sound clip, movie, 3D model, or whatever. It's generally synonymous with "file." Keeping one asset to a file is a good rule of thumb, but it is possible to store multiple assets in a file or for a single asset to have dependent assets, such that it spans several files.
opensource.com
A DAMS will typically provide something like a search interface combined with automatically collected metadata and user-assisted tagging. So, instead of having to remember where you put the file you need, you can find it by remembering things about it, such as when you created it, what part of the project it connects to, what's included in it, and so forth.
A good DAMS for 3D assets generally will also support associations between assets, including dependencies. For example, a 3D model asset may incorporate linked 3D models, textures, or other components. A really good system can discover these automatically by examining the links inside the asset file.
Version control for multimedia
Most software projects simply use their version control system (VCS) as a DAMS. The source tree provides a hierarchical scheme for locating files, which is entirely adequate for the rigid and static structure of program files that are intended to work together like parts of a machine.
opensource.com
Readers may be in familiar territory with VCS; you've likely heard of Subversion, Git, Bazaar, and Mercurial. However, these tools are terrible at handling multimedia assets. VCS documentation usually talks about this as if it were an intrinsic problem of multimedia files, but that's loser talk. The reality is that these applications simply aren't designed to cope with multimedia, so they fail badly at it.
Explaining exactly why that is (and how to really fix it) is a big enough subject to warrant a separate article, so, let's just talk about the consequences of it:
- You CANNOT merge multimedia files automatically
- That makes concurrent "merge-based" version control IMPOSSIBLE
- Therefore, all version control for multimedia should be on a "locking" basis
And with that, about 90% of the functionality of Subversion, Git, Bazaar, and Mercurial went right out the window and smashed on the pavement. Thanks anyway, guys!
- Multimedia files are HUGE
- Project structure is FLUID and LOOSELY DEFINED
- Intermediates can be EXPENSIVE
Typical VCSes simply have not been designed to cope with these facts. They all assume filesystem-type hierarchies of files are being stored, and most do not accommodate partial checkouts (or at least not easily; Subversion does do this better than the others). The relational, tagged organization of a DAMS is better suited to the fluidity of multimedia collections, and most have been designed with large file sizes in mind.
Searching for another option
We surveyed many different software packages, most of them not originally designed for our application. Some are used by other projects but don't seem optimal for ours. Others simply have issues with maintenance; they're either old and succumbing to bitrot (falling out of sync with current software) or still too immature to be used on an active project.
Some of the roads we didn't take:
- Blender-Aid is a Blender-refactoring tool that assumes you are using Subversion for the VCS. It was developed for use on the Blender Foundation's Sintel project, but unfortunately was not maintained after that project finished. When we tested it, we had to patch the source to get it to work with current Blender files (and that was several updates back). We submitted our patch, but no one was maintaining the project to accept it.
- Git is, of course, very popular with programmers. I'm often asked why we aren't on the Git bandwagon. It would be a small improvement over Subversion, but it's mostly a sideways move, making it not worth the trouble of migration.
- Git-LFS is an extension to Git that improves performance with large files.
- SparkleShare is a folder-sharing tool, based on Git, that's simple enough for artists without any VCS experience. It's used by the Tube project (Wires for Empathy).
- Boar can be seen as either a VCS or a backup system. It focuses on folder replication and easy handling of large multimedia files, although it doesn't help much with search or organization.
- Peragro goes to the other extreme, handling lots of small Blender assets well; in particular, it can look inside of Blender files to find the individual objects they contain. It's still very much in development, though.
- MediaWiki allows assets to be stored as "files" or "images," and we did use this approach for a short time during our pre-production work, but it's difficult to manage when asset storage is its primary use.
What we're using now
We are currently using a combination of Subversion, Trac, and Resource Space.
Subversion is a VCS to manage the Blender "source code" for our project. It is familiar and somewhat usable, although we're clearly overloading it. The Trac application provides a simple web-based view of the repository but doesn't preview multimedia content.
opensource.com
Resource Space is a DAMS with an excellent search and preview interface. Unfortunately, these programs don't really work together at all, so we must manually move files from one to the other when needed.
opensource.com
Trying a new TACTIC
In 2012, the Blender Foundation started a new project, called Gooseberry, which ostensibly had similar goals to ours, including producing a feature-length film with Blender. However, that has morphed into a series of shorts, and its solution for a studio backend was to go with a software-as-a-service platform called Blender Cloud. This was completely the opposite of what we wanted.
We looked into what options they had considered prior to that project and discovered TACTIC. TACTIC seems ideal, providing DAMS, VCS, and additional project management features, and it's licensed under the Eclipse Public License. Our biggest concern was that we weren't sure why Blender Foundation had opted against it. Possibly it was not under the EPL at that time (the package was relicensed about the same time as the Gooseberry project started) or simply too coupled to Maya (which it was originally developed to work with). Considering that Blender Foundation opted to build new, it may simply be a case of "not invented here."
opensource.com
However, as our intention is to invent as little as possible, TACTIC seems almost too good to be true. Our main infrastructure project for 2018 is to set up and test this system on our pilot project.
3 Comments