Introduction

event-routing-backends is developed as a pluggable application for the edx-platform. The code in this app hooks into the event-tracking app that is installed as a part of edx-platform. It provides new tracking backends and processors.

Features

Events that need to be transformed can be filtered by their names using either RegexFilter processor or NameWhitelist processor offered by the event-tracking library. Both these processors run in the main thread. NameWhitelist performs simple string comparisons and is therefore, faster.

In event_tracking_backends, two processors, namely CaliperProcessor and XApiProcessor can transform edX events into Caliper and xAPI format respectively. Events in Caliper format need to be passed through an additional processor named CaliperEnvelopeProcessor, after being transformed and before being routed.

EventsRouter backend runs these processors and then routes the transformed events (xAPI or Caliper format) to configured routers. It is configured as a nested backend (named xapi or caliper) of AsyncRoutingBackend (along with desired processors) in EVENT_TRACKING_BACKENDS configuration of the event-tracking library.

Execution

RegexFilter and NameWhitelist run synchronously in the main thread. Processors in xapi and caliper backends are executed asynchronously, with each backend executed as a separate celery task.

Routing of transformed events is also done asynchronously. Therefore, nested celery tasks are created, one for each configured router, to route events that have been transformed by xapi or caliper backends.

Retries

Once an event fails to transmit due to connection error, it is retried periodically for a finite number of times, with delay between each retry. Total number of retries and delay (in seconds) between each retry can be configured using plugin setting EVENT_ROUTING_BACKEND_MAX_RETRIES (default: 3) and EVENT_ROUTING_BACKEND_COUNTDOWN (default: 30), respectively. If it still fails to transmit, then the event is dropped unless it is configured to persist in the database.

Persistence

Event consumers may never want to lose certain events even after a brief failure of the connection or at the endpoint. List of these events can be specified in plugin setting EVENT_TRACKING_BACKENDS_BUSINESS_CRITICAL_EVENTS. Failed celery tasks for routing these events are persisted using edx-celeryutils package, once the retries have expired. edx-celeryutils also has commands for rerunning failed tasks and deleting old ones. Default list of events in EVENT_TRACKING_BACKENDS_BUSINESS_CRITICAL_EVENTS is:

  1. edx.course.enrollment.activated

  2. edx.course.enrollment.deactivated

  3. edx.course.grade.passed.first_time

Supported events and mapping of edx events onto xAPI and Caliper formats

List of supported edx events can be found in Supported_events along with their mapping onto xAPI and Caliper format.

Version information of mapping

Version of mapping of edx event onto xAPI or Caliper format, for each event, is included in the statement and event respectively. Version is a string of format “X.Y” where increment in X represents breaking changes and increment in Y represents addition of fields in the event json, both starting with zero.

In xAPI statement, version is in value of the key https://github.com/edx/event-routing-backends/blob/master/docs/xapi-extensions/eventVersion.rst in extensions of Context of the statement.

In Caliper event, version is in value of the key eventVersion in extensions of the event.

Installation

Install event routing backends library or add it to private requirements of your virtual environment ( requirements/private.txt ).

  1. Run pip install edx-event-routing-backends.

  2. Run migrations ( python manage.py lms migrate ).

  3. Restart LMS service and celery workers of edx-platform.

Configuration

Two types of configuration are needed for the plugin:

  1. Routers for routing transformed events to desired http endpoints.

  2. Backends for filtering and transformation of selected events into xapi or caliper format.

By default, both xapi and caliper backends are already configured along with filters that allow all the supported events. caliper backend is disabled by default and can be enabled by setting CALIPER_EVENTS_ENABLED to True in plugin settings.

Router configuration

Router(s) for each backend can be configured in django admin settings as follows:

  1. Navigate to http://localhost:18000/admin/event_routing_backends/routerconfiguration/add/

  2. Select Backend name (xapi or caliper).

  3. Add Route URL; the HTTP endpoint where events are to be received.

  4. Select Auth Scheme (Basic or Bearer or None). For Basic authentication, add username and password. For Bearer authentication, add Token.

  5. Add Configurations comprising of following configuration items as json:

    1. override_args: Accepts set of key:value pairs that will be added at the root level of the json of the event being routed. If the any of the keys already exist at the root level, their value will be overridden. Please note that for caliper backend, these changes will be made in the envelope.

    2. match_params: This can be used to filter events based on values of keys in the original edX events. Regular expressions can be used for values.

    3. headers: Additional headers can be specified here for caliper backend only.

A sample configuration for routing Caliper events having content organisation as edX AND course run is 2021 AND event name starts with problem OR event name contains video, with override arguments and additional headers:

{
    "override_args":{
        "sensor_id":"sensor@example.com"
    },
    "headers":{
        "test":"header"
    },
    "match_params":{
        "course_id":"^.*course-v.:edX\\+.*\\+2021.*$",
        "name":[
            "^problem.*",
            "video"
        ]
    }
}

A sample configuration for routing xAPI events if the enterprise is org_XYZ AND event name is edx.course.grade.passed.first_time OR edx.course.enrollment.activated:

{
    "match_params":{
        "enterprise_uuid":"org_XYZ",
        "name":[
            "edx.course.grade.passed.first_time",
            "edx.course.enrollment.activated"
        ]
    }
}

Backends configuration

By default, both caliper and xapi backends are configured with NameWhitelistProcessor that filters all the events currently supported. Users can override default backends to change filter type and name of the events to be filtered.

A sample override for caliper backend is presented below. Here we are allowing only enrollment, seek_video and edx.video.position.changed events to be filtered through RegexFilter to caliper backend.

EVENT_TRACKING_BACKENDS.update({
    'caliper': {
        'ENGINE': 'eventtracking.backends.async_routing.AsyncRoutingBackend',
        'OPTIONS': {
            'backend_name': 'caliper',
            'processors': [
                {
                    'ENGINE': 'eventtracking.processors.regex_filter.RegexFilter',
                    'OPTIONS': {
                        'filter_type': 'allowlist',
                        'regular_expressions': [
                            'edx.course.enrollment.*',
                            'seek_video',
                            'edx.video.position.changed'
                        ]
                    }
                }
            ],
            'backends': {
                'caliper': {
                    'ENGINE': 'event_routing_backends.backends.events_router.EventsRouter',
                    'OPTIONS': {
                        'processors': [
                            {
                                'ENGINE': 'event_routing_backends.processors.caliper.transformer_processor.CaliperProcessor',
                                'OPTIONS': {}
                            },
                            {
                                'ENGINE': 'event_routing_backends.processors.caliper.envelope_processor.CaliperEnvelopeProcessor',
                                'OPTIONS': {
                                    'sensor_id': 'http://example.com/sensors'
                                }
                            }
                        ],
                        'backend_name': 'caliper'
                    }
                }
            }
        }
    }
})

A sample override for xapi backend is presented below. Here we are allowing only enrollment, edx.course.grade.passed.first_time and edx.ui.lms.sequence.tab_selected events to be filtered through NameWhitelist to xapi backend.

EVENT_TRACKING_BACKENDS.update({
    'xapi': {
        'ENGINE': 'eventtracking.backends.async_routing.AsyncRoutingBackend',
        'OPTIONS': {
            'backend_name': 'xapi',
            'processors': [
                {
                    'ENGINE': 'eventtracking.processors.whitelist.NameWhitelistProcessor',
                    'OPTIONS': {
                        'whitelist': [
                            'edx.course.enrollment.activated',
                            'edx.course.enrollment.deactivated',
                            'edx.course.grade.passed.first_time',
                            'edx.ui.lms.sequence.tab_selected',
                        ]
                    }
                }
            ],
            'backends': {
                'xapi': {
                    'ENGINE': 'event_routing_backends.backends.events_router.EventsRouter',
                    'OPTIONS': {
                        'processors': [
                            {
                                'ENGINE': 'event_routing_backends.processors.xapi.transformer_processor.XApiProcessor',
                                'OPTIONS': {}
                            }
                        ],
                        'backend_name': 'xapi'
                    }
                }
            }
        }
    }
}