Why using XML Event Logs sucks using Splunk

Yesterday I had a discussion with a colleague if we should switch the indexing of Windows Eventlogs to XML. He mentioned that he was told that it’s faster, needs less data volume and language agnostic.

As I couldn’t imagine that something with the abbreviation “XML” in it could be something like “small” and “fast” I decided to do a test.

Interesting enough there is a blog article found at http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/ also stating that you would have an data reduction.Interesting enough there is a blog article found at http://blogs.splunk.com/2014/11/04/splunk-6-2-feature-overview-xml-event-logs/ also stating that you would have data reduction.

You start to collect XML Events with adding renderXml = 1 to the input stanza. When doing so the suppress_text = 1 is automatically set. Of course you could also omit the Eventlog message for your non-XML input and achieve the same volume reduction. Here I will keep the Eventlog message for the XML and non-XML scenario to ensure not comparing apples and oranges.

Index performance

The same dataset was indexed by the same forwarder on the same hardware..
let’s determine the time needed for indexing:

index=noxml OR index=xml
| stats count earliest(_indextime) as ite latest(_indextime) as itl by index
| eval timediff = itl-ite
| convert ctime(ite) ctime(itl)

1

Indexing the XML took 195 seconds vs. 153 seconds, 27,5% longer.

Size

Determine the indexsize:

| dbinsprect index=noxml OR index=XML | table index SizeOnDiskMB

2

XML needed 17,7% more storage.

Search performance

Running a noncomplex search, I can’t remember if I was more surprised that the XML search was more than 10X slower or that it showed a different result count.
I repeated the search 3 times to ensure the results are accurate.

index=noxml | stats count by EventCode – (fast mode enabled)

This search has completed and has returned 223 results by scanning 56,807 events in 2.923 seconds.
This search has completed and has returned 223 results by scanning 56,807 events in 2.91 seconds.
This search has completed and has returned 223 results by scanning 56,807 events in 2.888 seconds.

index=xml | stats count by EventCode – fast mode

This search has completed and has returned 106 results by scanning 56,807 events in 33.822 seconds.
This search has completed and has returned 106 results by scanning 56,807 events in 33.593 seconds.
This search has completed and has returned 106 results by scanning 56,807 events in 33.309 seconds.

All time ran into command.search.kv

3

Troubleshooting

It turns out the TA is not extracting all the information when using XML compared to using nonXML. This clearly looks like a bug and I need to follow-up on that.

4

5

I found a lot of events where the EventID is not extracted correctly from the XML..

index=xml OR index=noxml sourcetype=*application* RecordNumber=9172 | table EventCode sourcetype index _raw

6a

All tests was done on “latest an greatest” Splunk TA Windows v4.8.3 runing on Splunk Enterprise v6.5

Summary

Do never ever use XML rendering because of performance or expected data volume reduction. For now the only valid reason seems to be to overcome language issues..

Leave a Reply

Required fields are marked *.


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>