c++ - Optimising boost::python::object instances -

i started using boost's python library wrap parts of rather large c++ library.

quite chance, discovered every python object created boost python, @ least bigger size of python list.

simple types

using noddy_noddytype example c-python api docs, can quite expose new object type instance, noddy_noddyobject.

using raw c-python api, small , simple custom python object can be:-

>>> import noddy, sys  >>> print sys.getsizeof( noddy.noddy )   # pytypeobject size? 872 >>> print sys.getsizeof( noddy.noddy() ) # pyobject size 16 

and compare base object:-

>>> print sys.getsizeof( object ) 872 >>> print sys.getsizeof( object() ) 16 

that's expected, when expose noddy_noddyobject using boost python?

>>> print sys.getsizeof( noddy.noddy ) 904 >>> print sys.getsizeof( noddy.noddy() ) 80 

say what?!! that's not ideal... object instances 5 times bigger need be!

(this differ on non-64-bit machines.)

on boost python wiki, there's some explanation bloat:-

memory consumption

in general, wrapped c++ object corresponding python object size of:

• a new-style class (derived 'object' in python) instance plus
• the size required allow variable-length data in instance, plus
• the size of c++ object, plus
• the size of vtable pointer, plus pointer c++ object's instanceholder, plus
• zero or more bytes of padding required ensure instanceholder aligned.

you can see in boost/python/object/instance.hpp. python objects represented instance<value_holder<t> >, c++ class t.

but there notably no indication how reduce instance size, nor how use custom pytypeobject.

sequence types

so, in instance.hpp, can see boost::python::instance's use pyobject_var_head in base class's declaration. explain closer similarity in size of boost python-compiled noddy_noddyobject, of python list, or numpy.array:-

>>> print sys.getsizeof( list ) 872 >>> print sys.getsizeof( list() ) 72 >>> print sys.getsizeof( numpy.array ) 72 >>> print sys.getsizeof( numpy.array() ) 80 

(oh, that's interesting.. if i'm not mistaken, numpy.array size of minimal pyobject instance, smaller pytypeobject. wonder implications there..)

memory management

besides object size, concern led me here 1 of heap transactions. have c++ base class, comprised solely of memory management-related features: has new , delete operators; plenty of reference counting-related methods, , inherited hundred other c++ classes.

having once read c-python api docs on defining new types, thought new type appropriate here. defining new pytypeobject struct, , pointing members (tp_alloc, etc.) c++ class's operators , member functions, thought python interpreter call c++ member functions directly. derived class types have no functionality duplicated between typical boost::python::object instance , (already managed) c++ classes.

using bp::class_<> templates, figured both c++ code , python code separately managing memory class instances, seems potentially inefficient.

making new pytypeobject

by using new pytypeobject, think concerns resolved. memory footprint of simple objects smaller (i'd pleased if noddy_noddyobject down 24 bytes) , seems there far more flexibility how memory managed.

so, has else ever felt need define new pytypeobject, different boost python instances' default? how 1 go this?

by going larger base type, seems boost losing both efficiency , flexibility. it's more painful lose weight gain more.