Recently I read this set of posts:
A post from Konstantin Cherkasov claiming Glass to be 2000 times slower than the Sitecore API
A post from Kam showing Synthesis to be 3x slower than the Sitecore API
and finally a post from Richard Seal showing Fortis performance:
In this post I look more into what makes this a fairly useless test in the real world and debunk the idea that Glass Mapper is actually 2000x slower than the Sitecore API.
Background & Disclaimers
I fully support what all of the major ORM’s are trying to do in the Sitecore space. I have to some degree or other used all three in production website projects, I also know many of the individuals involved with the projects (sadly not yet met Kam).
I trust that Kam & Richard really know what they are talking about on their own frameworks, and I too have ran performance tests of my own using Glass, Fortis & Synthesis, so have been keeping up with the relative performance to the Sitecore API as well as in comparison to other ORM’s in the space.
I attempted to contact Konstantin before writing this with some optimisation tips (4 days ago) so he could amend / re-run his tests.
My tests are performed on consumer hardware on a VM (12GB RAM and 2 vCpu’s on a Macbook pro) – which puts it quite a way below production quality.
Finally – I must also state that the following is solely my own opinion.
I would also recommend you read my post here on Mappers and Wrappers as a back drop to allow you to see what is happening.
Ok – so onward – I would like to show what the basis of my version of the same code would look like (assuming the items collection was ‘warm’):
// Perform the initialisation somewhere else var sitecoreService = new SitecoreService("master"); Item item = items.First(); // Perform a warmup to ensure Glass has cached the expression to create that type of object var warmup = sitecoreService.Cast<Article>(item).Title; // Now perform the mapping items.Select(item => sitecoreService.Cast<Article>(item).Title).ToList();
So – with this in mind – I created this Gist
https://gist.github.com/cardinal252/1dbcc09e77b3ebeee84a. Please note that Glass by default performs a version count – so I added that to the sitecore side.
I performed this type of test on 2,600 items using the Glass test databases that we have (regular sitecore database) mapping the DisplayName (its the first common valid field) and found that the real world comparison (as I see the Gist above) would be about 4x (far from the 2000 times stated) the time that of the Sitecore API natively.
It is also worth noting that a cast of a single item 50,000 times took under 200ms.
I think in truth, if you are looking for quicker performance than that – you should not really be using an ORM in that area of code.
Why this test and comparisons are not useful
In truth – I would shoot a developer that attempted to manipulate 20,000 records worth of data through an ORM – just plain silly ;), but further than this. The ORM space as it stands today is vastly different in the Sitecore space, their functionality, how they achieve things is also vastly different, even against the Sitecore API. The fact Glass checks versions by default is a case in point. It can be disabled, but in general, the desirable behaviour in Glass is to be able to check for null. The other solutions provide this differently. But – further still – the whole ethos behind when / why / what ORM you choose is far more than ‘Blast 20,000 items at this and see how it compares to the Sitecore API’ just doesn’t provide meaningful results.
As all three will tell you, the real world use case for an ORM by definition (as all of the ORM providers agree) has a performance impact, Glass has performance impacts where the wrapping frameworks don’t, and gains in places where the wrapping frameworks don’t (caching / rendering etc). It, as always, is about choice. I have used all three of the major frameworks described here in production code and I still actively choose Glass for many reasons, I will gladly sell the pro’s of the mapping based approach, but this is not for everyone – that’s why there is not a clear winner in the debate.
For any bulk operation regardless of your choice of ORM, you will find that the Sitecore API should be the defacto choice.
Specific reasoning notes on the tests
Having read Konstantin’s, Richards and Kam’s post I found it would be worth writing a few notes on how and why I believe the test is both not so useful and inaccurate in places.
- During the load cycles using Glass, I invoked DotTrace on the test – it showed that close to 85% of the total test time was in Sitecore based assemblies. This is likely in part to be the cost of actually calling item.Field[“__DisplayName”] since it has a penalty in itself, as well as possibly things like LinkManager.GetUrl() etc.
- Version counts carry a cost – if you want to read / write blind then you can get better performance, but that is not how we would use the SC api in the real world. I am unsure at this juncture how the wrapping based solutions handle version counts (I suspect they are wrapped).
- I noted is that in Konstantin’s test – he performs 2 independent .Select()’s in the Glass example and only 1 in the second. Whilst I don’t believe this should cause a difference, the art to good testing is fair – like for like comparisons
- Konstantin looking at it has not read my post on How to correctly cast items to objects using Glass Mapper, which explains why you should never use item.GlassCast in the way that has been described. In short, you should not use item.GlassCast as a general rule without passing in the SitecoreService / SitecoreContext as the GlassCast extension will be forced to create it’s own (there is a debate as to whether this parameterless method should be removed altogether to avoid people making this mistake).
- Since Glass (to allow the use of more than one constructor) has to compile a lambda expression telling it how we effectively perform what is effectively ‘new Article();’ there will always be a performance hit for this compilation – similarly, Sitecore takes a hit on the first access of an item. For this reason, when performing tests (in particular with Glass), you would always provide a ‘warm up’ to enable these processes to fire since this is only ever taken the first time your application attempts to map that type.
- In Richard’s test – he has gone for only 50 iterations. The speed in which data comes back from Sitecore can vary wildly depending and will therefore wildly affect the tests in question. Whilst I doubt that it would make any difference to the overall result (Fortis and Synthesis are probably indeed quicker in this kind of operation), there is potential for a lot of variance between the three sets of results.
- Finally – when you are attempting to measure 50,000 operations that take place in 250ms, there are so many trivial things that take place that can skew your results severely, in that kind of territory – the movement of a mouse, a flash advert in your browser, a background clean up process can all cause a significant difference. In the more ‘real world’ test in my gist – the speed of the network IO, the load sql server is under etc DO prove a significant factor also since the item api on occasion will actively talk back to the db.
The end to end Glass performance against the raw Sitecore API, Glass’ source code actually has a LOT of like for like tests already in.
Since Glass is based on the Data Mapper pattern, a casting test is the worst relative test you can run against wrappers, this is a design decision. Mapping 20 properties and using only 1 is exactly what you shouldn’t do in your solution. In my experience, the varying frameworks level off quite a bit when you start reading the additional properties. This is exactly why I (unlike with wrapping frameworks) would never advocate the use of code generation in your Glass based solutions.
What has been shown here is a great example of (amongst other things) how to not understand the framework,how to bloat your model classes and exactly why you would not use an ORM for bulk operations of this kind.
You can see real world tests here (running live) on Glass’ performance page. These run in real time against the Glass database and show the display routines running 10x against sitecore’s own api. If you start adding caching (which I am hoping Mike will allow my field level caching mod too) – you will see more of where Glass’ advantages are.
In the real world, I don’t think the performance level of these three is actually something you would notice (trust me when I say that optimising the rendering pipeline in Sitecore can yield > 30% without changing anything else).