I've been looking at network rendering architectures for several years. My interest in network rendering has been primarily in conjunction with the computationally intensive process of rendering 3D animations from Discreet's 3D Studio MAX. 3D Studio MAX allows you to render animations using multiple computers. Each networked computer takes turns rendering individual frames of the animation. 3D Studio MAX is licensed such that a single, standard user license is all that's required to use the application for rendering on an unlimited number of networked computers running Windows NT.
The recent improvments to the network rendering feature in Adobe After Effects 4.1 Production Bundle and it's new license-free implementation have rekindled my interest in renderfarms.
Following are some observations based on my tests with After Effects 4.1 Production Bundle on Windows NT and Windows 98
Particularly interesting is a technique of using local watch folder rendering with multiple instances of After Effects running on a single multi-processor NT machine. It yielded almost 80% faster rendering! Finally SMP efficiency with After Effects!
I spend nearly half a day troubleshooting the network rendering setup. Some things I learned:
- First off all, in spite of what the on-line help says, UNC paths (file paths of the form \\machine\share ) don't work for watch folders. When selected using the File -> Watch Folder menu AE gives the error "After Effects error: invalid file location (volume name must be followed by a colon). (3::50)" After Effects 4.1 Online help says: "Note: All Windows machines monitoring the watched folder as a mapped network drive must map that drive using the same drive letter. If this is a problem, make sure the machine creating or collecting the project does not access the volume being watched as a mapped drive letter. For example, connect to the volume as \\network\watch instead of as X:\watch."
So just make sure you map the shared watch folder to the same drive ID everywhere.
It is also possible to specify a watch path on the command line when launching an After Effect Render Engine it may be that UNC paths work there. I've not test that.
- Make sure the actual watch folder is a subfolder of the network share even if the share is a subfolder to start with. (This is mentioned somewhere in the online help or read me.) The watch mechanism doesn't seem to work if its not set up this way.
- Collect Files command can't specify an absolute path for rendering output if "Change render output to" is selected. Results in error because it tries to create a folder called "(I:\AE Output 2)". It really should be possible to specify relative and absolute paths rather then just the pseudo-relative paths scheme they have (AE just supports a subfolder name which it puts in parenthesis telling AE to ignore the contents.)
You can re-render a collection in
a watch folder by deleting the output files, and editing the
It has been noted that the Render Engine install and the Production Bundle install are different and that each installs a complete After Effects program and set of subfolders including plug-ins etc. People often want to be able to run either the Render Engine or full program (w/ or w/o hardware lock) on the same machine. I did a DIFF on the two trees of the different installs. The only things different between the two installs (besides the obvious files like Help which are omitted in the Render Engine install) were the actual AfterFX.exe executable binary files and the PREF files.
I found that I could substitute either AfterFX.exe binary into a single After Effects folder and use the same set of plugins and support files for either binary. I also found After Effects doesn't like the name of binary to be changed from AfterFX.exe. (So no AfterFXPB.exe and AfterFXRE.exe.) I renamed the two different AfterFX.exe binaries AfterFX.exe.PB and AfterFX.exe.RE and made two batch files like this:
Copy AfterFX.exe.PB AfterFX.exe Start AfterFX
It is possible to make full use of multi-processing facilities on NT using the new multiple instance support in AE 4.1. You can launch multiple instances of After Effects and have them watch a local folder where you have collected projects for "network rendering". (Multiple instances of After Effects 4.1 can be started (on Windows only) by launching AE with the command "AfterFX.exe -m") Using this method I was able to get 100% processor loading much better than After Effects' typical 65%. This is the first time I've ever seen After Effects actually breaking a sweat on an SMP system.
I benchmarked the SpeedTest project recently made available by Atomic Power. (I modified the project to render to a compressed Targa sequence rather than a Sorenson QuickTime file.) A single instance of After Effects 4.1 completed the render on a dual-PII 400 system in 16 min. Using two instances of After Effects watching a local watch folder completed the render in 9 min! Same hardware. This benchmark is only a limited test in that it doesn't share any source footage.
I have since performed additional tests with projects that share source footage -- DV-AVI files -- and the results were similarly impressive performance improvements of about 70%.
Based on my initial results experimenting with the network rendering system in AE 4.1 I think it makes a lot of sense to investigate a renderfarm vs. ICE's solutions. Here are some of my thoughts on renderfarms and ICE rendering solutions.
The ICE solutions are quite expensive per system. ICE boards and software for After Effects range from $2000 to $8000 per system.
It makes me nervous that ICE haven't certified their ICE boards with any NT systems other than IBM and Intergraph workstations. Also since the majority of ICE's business seems to be oriented toward the Macintosh market I have some concerns about their ability to support their hardware on Windows NT systems. I have no information anecdotal or otherwise to support these concerns. ICE just doesn't have much of a track record with Wintel systems.
As a rendering architecture the ICE solutions are quite "brittle". The failure of a single system would be catastrophic to your organization's rendering capacity.
Reports indicate that ICE boards do perform quite well with ICE accelerated software.
The traffic jam effect problem: The ICE system doesn't help un-accelerated plugins at all. So a rendering project with only a single dependency on an un-accelerated effect can easily hold up the entire rendering pipeline and the benefits of the other acceleration may be diminished.
The one caveat with the network rendering solution is that your plugins need to be licensed for network use. Ideally they should be licensed in the same way After Effects is. (i.e. Licenses required for artists' workstations, but no license or hardware lock required for network rendering nodes.) This issue was discussed a bit on the AE mailing list a while back. I read that Atomic Power is licensing their plugins like this. I don't know about Final Effects and others though.
From what I've observed building a renderfarm really just becomes joint accounting and technical exercise of finding the balance between cost and rendering throughput. It's kind of an interesting exercise as such because almost all other issues go away. What's newest and fastest no longer matters. Sexy colored plastic cases don't matter. All that matters is cost and throughput.
Obviously the cost of administrative support for a renderfarm is not insignificant and needs to be factored into the overall cost of a renderfarm solution.
Renderfarms are pretty robust by nature of their architecture. If one system fails performance of the whole farm suffers proportionately, but the system keeps going. Individual rendernodes can easily be replaced even while the rest of the renderfarm is working.
Renderfarms also don't need to be symmetric with respect to processing power. As processors get cheaper and technology advances you just add new units representing the best price/performance at the time. But I think they generally do need to be symmetric with respect to memory configuration.
Adobe recommends against rendering to AE workstations. I haven't observed any problems rendering to a share off an AE workstation, but Adobe must have had some problems to make such a recommendation. And technically Microsoft has a licensing restriction on Windows NT Workstation being limited to 10 simultaneous connections. So you probably want to have an NT Server system administering any renderfarm larger than 10 machines. I don't think the NT Server system really needs to be very fancy and it could perhaps even be the same type of system as the rendernodes. Although I suspect a renderfarm server would ideally have more sophisticated storage systems such as tape drives or hot swappable drives etc. as the need arises. Again you don't want to become too dependent on any one device in the system lest it impact the system's overall reliability.
It looks like it is now possible to build cost effective network rendering nodes for under $1000. A year ago the sweet spot was more like $2000. This dramatic price reduction largely fueled by the availability of dual-processor Celeron systems and general price reductions in other components. Experience with 3D rendering with 3D Studio MAX has shown that dual-processor Celerons perform almost as well as similarly clocked PII and PIIIs. (See "The Celeron Myth" http://www.3dluvr.com/techz/celerymyth.htm ) After Effects should perform similarly. It may be possible to pull even more cost out of these systems by going with less conservative parts.
- Mac systems can be used for After Effects renderfarms either homogenously or heterogenously mixing Mac and Windows machines. Heterogenious renderfarms will require a multiplatform server such as Windows NT Server or one of third-party networking protocol stacks like DAVE or Miramar that can bridge the gap between Mac and Windows networking. Mac-based renderfarms represent a poor value relative to Windows-based renderfarms. The aledged benefits of Mac diminish when evalutating a system based on cost and performance for use as a renderfarm node. Mac-only shops will also often not have the technical expertise needed to administer a Windows-based renderfarm. So the costs to setup and administer a heterogenious or entirely Windows-based renderfarm for Mac-only studios will be higher.
- Gigabit Ethernet is now cheap and available. It is worth investigating to see if Gigabit yields anything in terms of cost/performance for the overall system.
These costs in my example generic dual-Celeron renderfarm node are pretty conservative. I deliberately looked at parts that had high MTBF and were designed for server type applications. I think you could pull a lot of cost out of such systems by going with cheaper parts as your confidence grows.
Q: Would I be better off with single processor systems if I will only get 70% efficiency from a dual processor system?
A: No. Since dual processor systems don't cost twice as much as single processor systems they don't have to run at 100% efficiency to pay their way. The incremental cost of the second Celeron in a dual Celeron system is only $80 to $150. So a dual processor system that runs 15% faster than a single processor system in a $1000 renderfarm node would pay easily pay for itself.
Fortunately, with the multiple instance trick that's possible with After Effects 4.1 you can get much better than 15% better performance from a dual processor system.
Copyright (c) 1999 Richard Lawler
- Richard Lawler
(richardl * at * well * dot * com)