Custom Logging with Event Hub and ADX

We have had customers who would like to develop a custom logging pattern but one of the challenges is they are focused on delivering solutions which add value to the business and building tools which can release the value in the logging data is not a primary concern.

In this sample we will look at a common custom logging pattern and show how you can take data from your custom event hub / ADX logging pattern and then use Turbo360 to visualize and present the data in a way that is usable by stakeholders who are not Azure experts and not developers.

The aim is to empower users to answer their own queries and reduce the support tickets sent to developers to check the logs.

Architecture

In this case you are using an Event Hub and using the data ingestion feature of ADX which will import events from the event hub into ADX so you can query them.

The sender of events can be any technology that is capable of writing data to the event hub.

Turbo360 is then querying ADX using the KQL queries you want to provide views of the logging data that matter to your scenario as shown below.

The key thing in this pattern is you would then have maximum flexibility to just log and can then decouple your presentation of the logs from the logging implementation.

Workflow

• The developer writes events to event hub from their chosen technology

• Azure Data Explorer is configured to ingest events from ADX

• The developer adds some KQL queries to Turbo360

• The L2 support user and business user can use Turbo360 to see what is happening in the interfaces

• The developer gets fewer support questions and tickets

• The L2 support user and business user are happier because they can self-service

Video

Parent Query

In this query I am looking at a table to find 2 events, one which represents the start of my process and one which represents the end of my process.

I will then join them together and return a row for each transaction which allows me to search via properties for the records I care about.

The KQL query for the parent can be anything you want, but I hope the below query can be a good example to help you get started.

Some key points about my query are:

1. In my table I have multiple events of different types so I can filter for just the ones I want

2. I need to promote a date field which will be used in Turbo360 for a time filter in the queries

3. One of the fields outputted by this query will be used to correlate from the parent query to the child queries

4. I can promote properties via the extends operator which will create fields you could search on in Turbo360

let startEvents = CustomBAM
| where BusinessProcess == "Mike-Test-BusinessProcess"
| where Transaction == "Mike-Test-Transaction"
| where Stage == "Started"
| extend CustomerName = Data.Name
| extend OrderID = Data.OrderID
| extend WorkflowName = Metadata.WorkflowName
| extend RunID = Metadata.RunID;
let endEvents = CustomBAM
| where IsTransactionComplete == true;
startEvents
| join kind=leftouter endEvents on $left.TransactionInstanceID == $right.TransactionInstanceID
| project StartTime=TimeStamp, EndTime=TimeStamp_1, TransactionInstanceID, BusinessProcess, Transaction, CustomerName, OrderID, WorkflowName, RunID, TransactionStatus=coalesce(TransactionResult1, TransactionResult)

Child Query

In the child query I am looking for a record which matches the transaction Id which I used as correlation from my parent query.

I am also looking for a specific stage name.

In my case I can reuse the same KQL query for each stage in Turbo360 and just change the stage name.  You could do similar things with your own queries.

CustomBAM
| where TransactionInstanceID == {TransactionInstanceID}
| where Stage == "Started"
| extend CustomerName = Data.Name
| extend OrderID = Data.OrderID
| extend WorkflowName = Metadata.WorkflowName
| extend RunID = Metadata.RunID