The NxuStream FAQ

What is NxuStream? NxuStream is a serialization library for the Ageia PhysX SDK. The Ageia PhysX SDK uses a data driven approach to create objects for the simulation. Rather than creating a rigid body and then calling numerous methods to configure it, instead all of the data needed to define the properties of a rigid body are placed into a data structure called a ‘descriptor’ and submitted in a single creation call.

The PhysX SDK likewise offers an API method that will save the current state of an object back out to this data structure so that a developer can preserve the state.

Since there are somewhere on the order of over 1,200 unique data items that can be represented in the Ageia PhysX SDK (this include every single bit flag and enumerated type) it takes a substantial development effort to read and write all of this state to and from single data resource.

NxuStream will take all or any sub-portion of the contents of the SDK and copy them to a collection of descriptors (referred to as an ‘NxuPhysicsCollection’) and then can save this collection to a number of file formats. It can also read that collection back into memory and, finally, instantiate it onto the SDK.

The uses for this include capturing a single object, an entire scene, or the whole SDK to a file for debugging purposes. These files can also be sent back to technical support to help diagnose problems.

Probably the most exciting use for NxuStream is to represent discrete data assets for your game. Combining the NxuStream data with authoring tools such as CreateDynamics, 3D Studio Max or Maya can be very powerful.

NxuPhysicsCollections can be instantiated at any location in the world. In this fashion you could have a single data asset that represents an object like a ‘tree’ or a ‘ragdoll’ and then place then instantiate them all over a game environment.

How do I use the library? NxuStream is delivered as a collection of source code that can compile against all versions of the PhysX SDK 2.4.0 and higher. Simply add the source code to your project (make sure your include paths are set up to point to the PhysX SDK) and build.

Only a single header file is needed to take advantage of the NxuStream API. Simply include ‘NXU_Helper.h’ and make a few calls.

The NxuStream library needs a small amount of persistent memory to keep track of name fields and the instantiation of mesh assets. To release this memory your application should, on exit or any time you do a full reset of the SDK, call ‘NXU::releasePersistentMemory’.

Why isn’t it just part of the SDK? NxuStream is a tool that, even though it is tightly bound to the SDK, is also independent of any specific version. The goal is to target past and future releases of the SDK with the same code base. This kind of backwards and forwards compatibility is critically important to help maintain data assets.

Another reason not to put NxuStream directly into the SDK is because it is a rather large code base and not everyone is going to want to use it in their product.

It is also quite possible that NxuStream would be revised independent of the SDK release schedule since new features such as COLLADA improvements would be unrelated to the SDK directly.

Why is it delivered as source code instead of a DLL or library? NxuStream is delivered as source code to provide the maximum flexibility for developers to incorporate it into their project. Of course it could be easily placed into a library or DLL by any individual developer if they wanted to.

Since NxuStream is entirely tools related code it is valuable for a developer to easily step into the source code when debugging or binding it to their own system. It is also quite likely that some developers will choose to place the NxuStream code into part of their tools chain as part of a game editor or used in plug-in components.

The NxuStream source code is entirely inside a namespace ‘NXU’ so it should have no conflicts with other source in an application.

Does NxuStream use the STL? NxuStream does not use the STL (Standard Template Library). For containers it simply uses the simple vector template ‘NxArray’ that is provided by the PhysX SDK.

However, even though NxuStream does not use the STL is still does quite a few memory allocations using the standard ‘new’ and ‘delete’ operator.

How much source code is in NxuStream? NxuStream comprises 42 source files, 22 CPP files and 20 header files.

This contains 33,308 lines total which breaks down as:

25,761 lines of code.

4,510 empty lines.

2,066 comment lines.

971 empty comment lines.

Why is there so much source code?

The quantity of source code simply reflects the amount of data that NxuStream has to represent. It must first be able to extract all data from the SDK and copy it into an ‘NxuPhysicsCollection’. Next it needs to be able to both read and write this data in a fault tolerant and human readable way. All memory ownership needs to be taken into account during this process. There is also code to instantiate the data back into the SDK and this too needs to be fault tolerant and keep memory ownership into account.

Finally, the biggest reason for the quantity of source code is having to support both COLLADA and NxuStream XML; this requires not only an XML parser but also an interpreter that can filter that XML data and convert it from ASCII into a binary representation.

What file formats does NxuStream support? NxuStream supports three file formats, XML, BINARY, and COLLADA. The XML and binary versions are a true reflective object model of all of the PhysX SDK data descriptors. What this means is that not only is all data represented but also, wherever possible, the name of each data item in the XML is identical to the name found in the data structure. For example if a data structure would have a member variable called ‘globalPose’ then you can be sure the XML file will place it inside an element called ‘globalPose’. This one to one correspondence between the XML file and the SDK makes it much easier to visually debug problems simply by inspecting the file.

The COLLADA support is based on the 1.4.1 schema and is limited to representing physics data only. NxuStream does not do anything with the graphics portion of a COLLADA file. If you load a COLLADA file with both graphics and physics NxuStream will only concern itself with the physics data. A developer can bind physics objects with graphics or game objects using either user property fields or names.

What is COLLADA? COLLADA is an industry standard 3d asset interchange format. It is the first such format to provide support for physics content. When using COLLADA, unlike the reflective formats (XML and binary), not all concepts in the Ageia PhysX SDK will be preserved. Support is provided for this format to ease migration from various 3rd party authoring solutions that use COLLDA.

Why are there so many formats? Each of these three formats serves a particular need.

COLLADA is useful when working with 3rd party authoring tools that recognize and support this physics authoring in this format.

NxuStream XML is useful when authoring data assets that require complete control over every feature in the PhysX SDK. It is especially useful for debugging and troubleshooting problems since the XML can easily be visually inspected.

NxuStream Binary is useful for final production assets for high speed data loading. The binary format supports ‘cooked’ data formats that can allow large assets to be loaded into the SDK nearly instantaneously. A drawback of the binary file format is that it is not forwards or backwards compatible so it is best to keep the XML version as a reference to ease migration across SDK releases.

Why aren’t binary files backwards compatible? Since the binary version of the data is designed to loaded and saved at high speed directly to and from various descriptor data structures it is extremely challenging to maintain forwards and backwards compatibility when those data structures change across SDK versions. Additionally any time new features are added to NxuStream itself it would invalidate the previous data assets.

The XML version is much more flexible and robust and does not suffer from these same problems. It is best to keep source data in XML and only burn it into a binary cooked file as a production process.

What is ‘cooked’ data? ‘Cooked’ data represents triangles meshes and convex hulls in a format that the PhysX SDK can directly operate on. Before a triangle mesh or convex hull can be submitted to the SDK it must first be ‘cooked’ which involves a lot of operations to manipulate the source data into an optimal representation for high speed collision detection. Internally various data structures are built as well as general cleanup. Once data has been ‘cooked’ it can be loaded into the SDK almost instantaneously.

Is NxuStream multiplatform? Yes, NxuStream has been tested on Windows, Unix, and an XBOX-360.

What XML parser does NxuStream use? NxuStream uses the ‘TinyXML’ parser to load XML data assets. TinyXML, as the name implies, is very tiny. It is a stable, robust, and fast DOM XML parser in only a few thousand lines of code. This implementation of TinyXML has been wrapped into a namespace and redirects file IO to support reading to and from a buffer in memory.

What is the purpose of the individual source files provided?

NXU_Helper.cpp and NXU_Helper.h

This is the main public interface to NxuStream. An application should only ever need to include this single header file to have access to all of the functionality needed.

NXU_Asc2Bin.cpp and NXU_Asc2Bin.h

This is a utility that helps convert ASCII data into a binary representation.

NXU_BinaryStream.cpp and NXU_BinaryStream.h

This is the implementation of the stream class to read and write the binary version of the data. NxuStream uses a pure virtual interface so that reading and writing both binary and the XML versions is largely the same code.

NXU_ColladaExport.cpp and NXU_ColladaExport.h

This code saves out the contents of an NxuPhysicsCollection to a COLLADA 1.4.1 physics file.

NXU_ColladaImport.cpp NXU_ColladaImport.h

This source parses a COLLADA file and extracts all of the relevant physics content. This physics data is then copied to an NxuPhysicsCollection.

NXU_Cooking.cpp and NXU_Cooking.h

This is a utility that invokes the version controlled cooking library.

NXU_File.cpp and NXU_File.h

This is a utility that can vector standard file IO routines to a buffer in memory if needed.

NXU_Geometry.cpp and NXU_Geometry.h

This is a utility that can build a renderable mesh from primitives such as box, sphere, or capsule.

NXU_Hull.cpp and NXU_Hull.h

This is a utility that generates a convex hull from a point cloud.

NXU_PhysicsExport.cpp and NXU_PhysicsExport.h

This source extracts the contents of individual SDK objects into NxuStream descriptors that can be added to an NxuPhysicsCollection.

NXU_PhysicsImport.cpp and NXU_PhysicsImport.h

This source invokes the stream read routines to load a physics collection into memory.

NXU_PhysicsInstantiator.cpp and NXU_PhysicsInstantiator.h

This source instantiates the contents of an NxuPhysicsCollection into the PhysX SDK.

NXU_schemaparse.cpp and NXU_schemaparse.h

This is an internal utility to read the NxuStream XML schema and produce debugging data.

NXU_Stream.cpp and NXU_Stream.h

This is the main streaming code that handles both reading and writing all of the contents of an NxuPhysicsCollection

NXU_StreamFactory.cpp and NXU_StreamFactory.h

This is a small utility to retrieve the appropriate input or output stream interface based on a particular file type.

NXU_Streaming.cpp and NXU_Streaming.h

This is a small utility to stream cooked data to and from blocks in memory.

NXU_String.cpp and NXU_String.h

This is a small utility that keeps track of persistent strings and asset bindings.

NXU_tinystr.cpp and NXU_tinystr.h

This source provides string support for TinyXML

NXU_tinyxml.cpp and NXU_tinyxml.h

This source contains the core of the TinyXML data structures and API.

NXU_tinyxmlerror.cpp

This source is used by TinyXML to keep track of the current error state.

NXU_tinyxmlparser.cpp

This source contains the core XML parsing routines used by TinyXML

NXU_XMLStream.cpp and NXU_XMLStream.h

This source provides support to read and write the contents of an NxuStream XML data file.