Tables with tables in tables – ShiVa Engine

Tables with tables in tables

ShiVa is a C++ engine with a Lua scripting interface. While this works great for calling API functions, it does have its pitfalls when it comes to organizing and storing data. Since tables are the main (in fact, the only) data structuring mechanism in Lua, these pitfalls become especially apparent when trying to combine ShiVa tables with Lua tables.

ShiVa tables

ShiVa tables use the table.* API. If you want to store vectors, arrays, lookups, bit-/boolfields or similar, this is your go-to data structure. Tables translate perfectly to C++ code.

Common usage

Tables are typically AI member variables (this.*), since you typically want to store data in them over multiple frames. To improve performance, it is a good idea to always reserve some memory for the table if you have an idea how big the table will be, since you won't have to pay the cost of memory allocation during table.add() if the space required is already allocated:

  1. local lt1 = this.t1 ( )
  2. local lt2 = this.t2 ( )
  3. local lt3 = this.t3 ( )
  4.  
  5. table.reserve ( lt1, 3 )
  6. table.reserve ( lt2, 3 )
  7. table.reserve ( lt3, 3 )

To add items to a table, you can use the table.add() function, and retrieve data using table.get():

  1. function tabletest.standardTables ( hTable )
  2.  
  3. local tsize = table.getSize ( hTable )
  4.  
  5. for i=1, tsize do
  6. log.message ( "Content: " ..table.getAt ( hTable, i-1 ) )
  7. end
  8.  
  9. end
  10.  
  11.  
  12. table.add ( lt1, 1 ); table.add ( lt1, 2 ); table.add ( lt1, 3 )
  13. table.add ( lt2, 4 ); table.add ( lt2, 5 ); table.add ( lt2, 6 )
  14.  
  15. this.standardTables ( lt1 )
  16. this.standardTables ( this.t1 ( ) )
  17.  
  18. -- OUTPUT: (2x) -----------------
  19. --Content: 1
  20. --Content: 2
  21. --Content: 3

As you can see above, it makes no difference whether you call the local variable "lt1" or the member function "this.t1()", since lt1 is a reference to the original table, not a copy. All changes you make to the reference will actually be made no the table itself.

Table nesting

ShiVa tables can be nested in order to create multidimensional tables/arrays:

  1. table.add ( lt2, 4 ); table.add ( lt2, 5 ); table.add ( lt2, 6 )
  2. table.add ( lt3, 7 ); table.add ( lt3, 8 ); table.add ( lt3, 9 )
  3.  
  4. -- nest lt2 and lt3 into lt1
  5. table.add ( lt1, lt2 ); table.add ( lt1, lt3 )
  6.  
  7.  
  8. function tabletest.nestedTables ( hTable, nIndex )
  9.  
  10. local sub = table.getAt ( hTable, nIndex )
  11. local tsize = table.getSize ( sub1 )
  12.  
  13. for i=1, tsize do
  14. log.message ( "Sub " ..nIndex .." Content: " ..table.getAt ( sub, i-1 ) )
  15. end
  16.  
  17. end
  18.  
  19.  
  20. this.nestedTables ( lt1, 1 )
  21. this.nestedTables ( this.t1 ( ), 1 )
  22.  
  23. -- OUTPUT: (2x) -----------------
  24. --Sub 1 Content: 7
  25. --Sub 1 Content: 8
  26. --Sub 1 Content: 9

Again, working with the reference or the member variable itself makes no difference.

Mixed value types

ShiVa allows you to store variables of perceived different type inside tables. code like this is legal...

  1. table.add ( lt2, 4 ); table.add ( lt2, 5 ); table.add ( lt2, 6 )
  2.  
  3. table.add ( lt1, "a string" )
  4. table.add ( lt1, lt2 )
  5. table.add ( lt1, 5 )
  6.  
  7. this.standardTables ( lt1 ) -- ERROR at #2
  8. this.nestedTables ( this.t1 ( ), 1 )

... but will throw if you are not careful with your evaluation function. Passing lt1 to standardTables() will result in an error because you cannot log.message() a table (index 1, value lt2) directly, only its contents. If you plan on nesting tables and mixing values, do not mix tables and values on the same "level":

BAD:

+ table
-- number
-+ subtable
--- subtablevalue1
--- subtablevalue2
--- subtablevalue3
-- string


GOOD:

+ table
-+ subtable1
--- number
-+ subtable2
--- subtablevalue1
--- subtablevalue2
--- subtablevalue3
-+ subtable3
--- string

Storing ShiVa objects

ShiVa tables can be used to store not only strings, numbers and tables, but also ShiVa objects.

  1. local lt = this.t1 ( ); table.reserve ( lt, 3 )
  2. local hS = application.getCurrentUserScene ( )
  3.  
  4. local m1 = scene.getTaggedObject ( hS, "box1" ) -- scene object
  5. local m2 = scene.createRuntimeObject ( hS, "" ); scene.setObjectTag ( hS, m2, "box2" ) -- runtime
  6. table.add ( lt, m1 ); table.add ( lt, m2 )
  7.  
  8. log.message ( "ShiVa table object tag 0: " ..scene.getObjectTag ( hS, table.getAt ( lt, 0 ) ) )
  9. log.message ( "ShiVa table object tag 1: " ..scene.getObjectTag ( hS, table.getAt ( lt, 1 ) ) )

As long as the object exists (not nil), it does not matter whether the object was placed in the scene (m1) or generated at runtime (m2).

Lua tables

Lua tables should never be your first choice for storing and arranging data in ShiVa. If you decide on using Lua tables, you simultaneously decide against being able to translate your code to C++ for extra speed benefits. In essence, you have to weigh performance (table API, C++) against coding convenience (Lua tables).

Lua tables use the curly braces {} syntax. Tables are the main (in fact, the only) data structuring mechanism in Lua. Tables are used to represent ordinary arrays, symbol tables, sets, records, queues, and other data structures. The table type implements associative arrays. An associative array is an array that can be indexed not only with numbers, but also with strings or any other value of the language, except nil. Moreover, tables have no fixed size; you can add as many elements as you want to a table dynamically.

Tables are initialized in Lua with curly braces:

  1. -- empty table
  2. a = {}

Lua table indices start at 1, not at 0 like in the ShiVa table API. Lua tables can contain numbers, strings, a mixture of both, ...

  1. -- number table
  2. b = {7, 8, 9}
  3. log.message(b[1]) --> 7
  4. log.message(b[7]) --> nil
  5.  
  6. -- mixed table
  7. c = {7, "eight", 9}
  8. log.message(c[2]) --> eight

... or even other tables. Accessing them is easy as well:

  1. -- tables in table
  2. d = { {1, 2, 3}, {4, 5, 6}, {7, 8, 9} }
  3. log.message ( "table value at d 2,2 index: " ..d[2][2] ) --> table value at d 2,2 index: 5

Unlike Editor Lua, the Runtime Lua does not support the # operator to count the elements of a table. You will have to write your own count() function, which loops through all elements of a table:

  1. days = {"Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"}
  2.  
  3. function tcount (luatable)
  4. local c = 0
  5. for i,v in ipairs(luatable) do
  6. c = c+1
  7. end
  8. return c
  9. end
  10.  
  11. log.message ( "count: " ..tcount(days) )

Global tables

Lua variables and functions you want to access from any AIModel are stored in a global Lua table named _G. This table is not static, you can add your own entries through the rawset() function. To make it easy to use, you could write a manager function like this:

  1. function tabletest.declare (name, initval)
  2. rawset(_G, name, initval or false)
  3. end

Using a global table would look something like this:

  1. -- true global table
  2. this.declare ( "gtable", { 7,8,9 } )
  3. log.message ( "gtable[2]: " ..gtable[2] ) --> 8

Nested global tables are of course possible too.

Passing Lua tables as argument

Lua tables can be passed as arguments to AI member functions:

  1. function tabletest.read2 ( luatable )
  2.  
  3. log.message ( "read2: " ..luatable[2] )
  4.  
  5. end
  6.  
  7. local lt = { 1,2,3 }
  8. gt = { 4,5,6 }
  9. -- nesting
  10. local mt = { lt, gt }
  11.  
  12. this.read2 ( gt )
  13. this.read2 ( lt )
  14. this.read2 ( mt[2] )

However this is limited to direct member function calls. Using Lua tables in ShiVa API functions will cause errors:

  1. local lt = { 1,2,3 }
  2. user.postEvent ( this.getUser ( ), 2, "tabletest", "onReceive", lt ) -- error!

In this case, it also does not matter whether the Lua table is declared globally or not. Passing a Lua table as argument to an API function will most likely fail.

Lua tables as object storage

Lua tables are made to store Lua types, namely bool, string, number, function, nil and table. Things like objects, scenes etc are not strings or numbers, their type is "userdata":

  1. local m1 = scene.getTaggedObject ( hS, "box1" )
  2. log.message ( m1 ) -- COMMON_box111
  3. log.message ( type(m1) ) -- userdata
  4. log.message ( tostring(m1) ) -- userdata: 0000000000000003

It is possible to temporarily store userdata objects inside Lua tables...

  1. this.declare ( "otable" )
  2. local m1 = scene.getTaggedObject ( hS, "box1" )
  3. local m2 = scene.createRuntimeObject ( hS, "" ); scene.setObjectTag ( hS, m2, "box2" )
  4. otable = { m1, m2 }
  5. -- will work, but only inside this script
  6. log.message ( "Lua global table object tag 1: " ..scene.getObjectTag ( hS, otable[1] ) )
  7. log.message ( "Lua global table object tag 2: " ..scene.getObjectTag ( hS, otable[2] ) )

.. but these object references will not be valid in another script/function/AIModel:

  1. if otable[1] == nil then
  2. log.warning ( "NIL" ) -- this will never be called
  3. end
  4.  
  5. -- Errors/crashes
  6. log.message ( "Lua global table object tag 1: " ..scene.getObjectTag ( hS, otable[1] ) )
  7. log.message ( "Lua global table object tag 2: " ..scene.getObjectTag ( hS, otable[2] ) )

What makes this so dangerous is that "otable[1]" in the example above will not be nil, so there is no proper way to safeguard against this type of undefined behaviour - avoid this pattern at all cost, as it easily leads to crashes.

Combining tables

The entire next section can be summed up in a single sentence: "Don't mix Lua and ShiVa tables if you don't have to!" Most of the time, it will not work, or behave in ways you would not expect. That said, let's have a look.

Lua table in ShiVa table

Does not work:

  1. local lt = { 1,2,3 }
  2. -- store lua table in shiva table
  3. local st1 = this.t1 ( )
  4.  
  5. table.add ( st1, lt )
  6. local r1 = table.getAt ( st1, 0 )
  7. log.message ( "r1: " ..r1[2] ) -- attempt to index local `r1' (a nil value)

Adding globally declared tables also does not work. "table.add()" will always add "nil".

Local ShiVa table in Lua table

Does not work:

  1. -- store shiva table in lua table
  2. local st = table.newInstance ( ); table.reserve ( st, 3 ); -- or non-local: makes no difference!
  3. table.add ( st, 1 ); table.add ( st, 2 ); table.add ( st, 3 );
  4. gtable["_st"] = st
  5. log.message ( "gtable['_st'][2][1]: " ..gtable["_st"][2][1] ) -- attempt to index field `_st' (a userdata value)

ShiVa AI Member table in Lua table

Works, but only in the same script file/function.

  1. -- get member table
  2. local st2 = this.t2 ( )
  3. table.reserve ( st2, 3 )
  4. table.add ( st2, 4 )
  5. table.add ( st2, 5 )
  6. table.add ( st2, 6 )
  7. -- store member table in mt{}
  8. mt["_st2"] = this.t2 ( )
  9. log.message ( 'mt["_st2"][1]: ' .. table.getAt ( mt["_st2"], 0 ) ) -- works, but only in the same script

As soon as you declare mt{} globally and call it from another script file or AI, the reference to the AI Member table will be invalid.

Hashtables

Apart from using indices in tables, you can also refer to items by a key string. This concept is implemented in ShiVa's hashtable API. Its design and limitations are similar to tables.

Common usage

The ".add()" function works in analog to tables. Since hashtables do not rely on contiguous memory, there is no reserve() command.

  1. local lht1 = this.ht1 ( )
  2. local lht2 = this.ht2 ( )
  3. local lht3 = this.ht3 ( )
  4.  
  5. hashtable.add ( lht1, "first", 1 ); hashtable.add ( lht1, "second", 2 ); hashtable.add ( lht1, "third", 3 )
  6. hashtable.add ( lht2, "first", 3 ); hashtable.add ( lht2, "second", 5 ); hashtable.add ( lht2, "third", 6 )
  7.  
  8. log.message ( "HT standard this(): " ..hashtable.get ( this.ht2 ( ), "second" ) )
  9. log.message ( "HT standard local: " ..hashtable.get ( lht2, "second" ) )

Nesting

Hashtables can be nested. Extending the sample from above:

  1. hashtable.add ( lht3, "metahash1", lht1 )
  2. hashtable.add ( lht3, "metahash2", lht2 )
  3.  
  4. log.message ( "HT3->HT2->first: " ..hashtable.get ( hashtable.get ( this.ht3 ( ), "metahash2" ), "first" ) )

Mixing tables

The same limitations for mixing Lua tables and ShiVa tables apply to Lua tables and Hashtables:

  1. -- storing Lua tables in hashtables
  2. local mt = { 7,8,9 }
  3. hashtable.add ( lht3, "luatable", mt )
  4. local r2 = hashtable.get ( this.ht3 ( ), "luatable" )
  5. --log.message ( "Lua table in hashtable: " ..r2[1] ) -- attempt to index local `r2' (a nil value)
  1. -- storing ShiVa HT in Lua table
  2. this.declare ( "hts", {this.ht1 ( ), this.ht2 ( )} )
  3. log.message ( "ShiVa HT in Lua table: " ..hashtable.get ( hts[2], "first" ) ) --only works in this function

A better pattern

Instead of trying to store incompatible datatypes into unsuitable containers, you should instead use datatypes that both languages, C++ and Lua, understand easily: strings and numbers. It is no coincidence that ShiVa does not have datatypes for textures, AIModels, sounds or scenes, and instead refers to them via identifier strings in a get/set pattern:

  1. application.setCurrentUserScene ( "UniqueSceneIDString" )
  2. -- or
  3. hud.getComponent ( hUser, "myHUD.myComponent" )
  4. -- etc.

No scene object and no HUD object get passed around in the above example, only strings. Therefor, the proper way to "store" a ShiVa table inside a Lua table is also by using a unique identifying string, and then perform a lookup on the identifier:

  1. -- file one
  2. gtable["_st2"] = "t2"
  3.  
  4. -- file two
  5. local v = gtable["_st2"]
  6. if v == "t2" then
  7. log.message ( 'gtable -> t2: ' .. table.getAt ( this.t2 ( ) , 0 ) )
  8. else
  9. log.warning ( "No valid expression!" )
  10. end