Unindexed Entity properties with Bulk Loader

Bulk Loader is a really nice feature of Google App Engine. The only thing I could not find is how to upload unindexed properties.

Fortunately, there's a post_import_function hook that can be run on every entity just before it gets uploaded. The code is really just a couple lines:

(goes into something like postimport.py)

from google.appengine.api import datastore

_UNINDEXED_PROPS = {
  'MyModel': ['prop1', 'prop2', 'prop3'],
  'MyOtherModel': ['some_prop']
}

def unindex_properties(input_dict, entity, bulkload_state_copy):
"""Runs on after every entity import and sets correct
unindexed properties.
"""
if isinstance(entity, datastore.Entity):
  kind = entity.kind()
  if kind in _UNINDEXED_PROPS:
    props = _UNINDEXED_PROPS[kind]
    entity.set_unindexed_properties(props)
return entity

And here's a sample bulkloader.yaml snippet:

python_preamble:
- import: base64
- import: re
- import: google.appengine.ext.bulkload.transform
- import: google.appengine.ext.bulkload.bulkloader_wizard
- import: google.appengine.ext.db
- import: google.appengine.api.datastore
- import: google.appengine.api.users
- import: postimport

transformers:
- kind: MyModel
  connector: simplexml
  connector_options:
    xpath_to_nodes: /whatever
    style: element_centric
  post_import_function: postimport.unindex_properties
  property_map:
  - ...props definitions...

Gist: https://gist.github.com/3351222