Lucinq Features – Date range searching

Pulling items by date or date range out of Lucene is quite a useful process. To do it accurately, we need to store the date values in a field that can be parsed appropriately. In Lucinq, our fluent Lucene wrapper, we currently achieve this by converting the DateTime value into an integer using the ticks property. This has a resolution down to 100ns which should be enough for anyones needs!

Using this, it allows us to create a range based search by comparing the ticks values between two dates. Then, for Lucene this is a simple integer comparison (taken directly from our test suite):

LuceneSearch luceneSearch = new LuceneSearch(IndexDirectory);

IQueryBuilder queryBuilder = new QueryBuilder();
DateTime february = DateTime.Parse("01/02/2013");
DateTime end = DateTime.Parse("28/02/2013");

queryBuilder.Setup(
	x => x.WildCard(BBCFields.Description, "food", Matches.Always),
	x => x.Filter(DateRangeFilter.Filter(BBCFields.PublishDateObject, february, end))
);

ILuceneSearchResult result = luceneSearch.Execute(queryBuilder);
List<NewsArticle> data = Mapper.Map<List<Document>, List<NewsArticle>>(result.GetTopItems());

This is similar to the way you’d think about doing it in SQL. Behind the scenes, SQL Server stores datetime values with very high precision.

Its worth noting that we’ve used the date based range as a filter in this example. We can also apply a date range directly and have it work like so:

LuceneSearch luceneSearch = new LuceneSearch(IndexDirectory);

IQueryBuilder queryBuilder = new QueryBuilder();
DateTime month = DateTime.Parse("01/02/2013");

queryBuilder.Setup(
	x => x.DateRange(BBCFields.PublishDateObject, month, month.AddDays(28)),
	x => x.WildCard(BBCFields.Description, "food")
);

ILuceneSearchResult result = luceneSearch.Execute(queryBuilder);
List<NewsArticle> data = Mapper.Map<List<Document>, List<NewsArticle>>(result.GetTopItems());

I think this is a good example of picking the correct field type for the job. By avoiding using a string we gain a lot more relevant operations without issues in casting the data. A case study for this would be a project I completed some time ago where cooking times needed to act as filters for a search. We were unable to achieve consistent results until the cooking time was stored as an integer value rather than a string. Which is something you would expect in code but potentially neglect in a search index.

Resources

Sitecore conventions – build your own pipelines

This post is a follow up the post on using workflow I wrote a little while ago. You can read it here.

When you look deeper under the hood you’ll see that Sitecore does quite a lot of things via pipelines. A brief scan of the web.config will show a wide selection ranging from logging in, begin request, rendering fields and so on. These pipelines make it very simple to define a process and to support customisation and patching of that process. As a consequence its something that can be quite useful to include in our own code when we need to perform our own custom steps.

In the previous article, I wrote about how I was talking with another developer about sending newsletters from Sitecore. The original third party had done this as a publish step and checkbox. So the editor would check the box, publish and then the custom code would uncheck the box, save the item and then send the email. However I discussed how there were a number of issues with this approach. In this article, we’ll build on this and see how using pipelines can help.

Continue reading

Life Through A Lens – Unit Testing your .net code using Glass Mapper

So now I have gotten my first couple of Sitecore User Group presentations out of the way – I thought I would actually continue on my journey of showing how I use Glass Mapper on the Sitecore platform in real world application.

As you will see from other posts I write – I am very much a TDD advocate. I find it helps focus my mind much better than simply writing code ad-hoc. Also – I hate testing, but love coding – so what better way to get my testing done than coding.

In choosing an ORM, I believe this should be one of your primary criteria since you are going to take some sort of performance hit with it being in the way and for me, trying to justify as just being ‘prettier’ code doesn’t really cut it when you are presenting it to a wider team.

With this in mind, I thought I would kinda tie these two together and look at how I use Glass to perform unit testing on *MY* code.

Continue reading

Unit Testing – Thoughts from an opinioniated developer

Getting started

Unit testing is one of those things that we know we should be doing. However, time and project pressures often result in various reasons why many don’t do it.

Having worked and consulted in a variety of teams I often find myself dealing with situations like this by asking the following questions:

  1. How do you know your software is working?
  2. How do you know if your software is broken?
  3. How do you know if someone elses change won’t break your hard work?

Generally, these are the three important questions that I try to answer when writing my own unit tests. Whether for Sitecore or not. The content of these tests is to try and prove it. Initially, writing a quality unit test can be hard. A quick search on Google will tell you that a unit test should be small, repeatable, focused and relevant.

That’s a useful guideline but we need a bit more information to get us started. To reinforce this, lets take an aside and consider a project I worked on for a major UK supermarket retailer.

In this project, the customer mandated 95% code coverage across the entire codebase. So the developers were focused on reaching this percentage and would be testing constructors and the like. More importantly they were jumping through hoops to test all their conditional branching, magic flags and so on. They were doing this
as they had code colouring turned on and could see which lines weren’t covered.

So, what to take away from this? Well firstly they weren’t writing their code to be testable. This should be our first rule:

Write your code to be testable

Next lets consider how they were getting code coverage. They were looking at how to test each individual line rather than consider what the code as a whole was attempting to do. If we stand back, we can see how the code is expected to operate under normal conditions. So that’s our second rule:

Test your code under normal conditions with expected parameters

By now we should have an idea of what our third rule should be! Now we start to think about how our code can break. What if a parameter is null? What if we have an unexpected value somewhere? We need to test the code under unexpected conditions and understand where and how it can break.

Test your code under unexpected conditions and with worse case scenarios

This is a very useful step. If for nothing else its something that you might have to cover at code review. It also means you can diagnose a production system more easily because in handling your unexpected conditions you have the opportunity to log, throw custom exceptions, raise alarm bells (and so on).

At this point, you should be in a position to have quite a reasonable level of code coverage but more importantly the tests in place are relevant, focused and they should be repeatable.

So lets consider the next rule of unit testing:

What am I actually testing?

No code sits in isolation. We access databases, save to files on disk, send messages, create pictures and so on. The list is endless. Because of this, we are always going to be calling third party code and integrating with third party systems.

So should we test them? No. Should we test our integration with them? Possibly. Should we test our calls to the integration layer? Yes. Provided we have abstracted our integration behind our own interfaces (such as a repository object or service object).

Encapsulate what ‘varies’ behind an interface, code to that interface and then write unit tests for code that depends on that interface according to the rules above. And if you only get 85% code coverage well the remaining 10% is a judgement call between you and your customer/employer!

So how do we write testable code?

Well written code is something that takes time and experience to produce. To start with testable code we could go back and cover writing code according to SOLID principles. Lets put that at the back of our minds for a moment and consider it in more abstract terms.

One method of writing testable code is to write the test first. This puts us into the mind set of thinking about how the contract will be used by calling code. Here we have an idea of our parameters, return types and the like. In fact many unit test advocates would argue that this is the only way to do it. Of course in reality it depends on where you work and how defined your designs/stories are.

Another tool on our development shelf is to consider how we might build or refactor the code internally. One example is switching on magic strings or enum values. Can we replace this with polymorphic calls and remove conditionals? If so, we can make our test cases easier to produce because we don’t have to setup our unit test to have mocked data that handles both sides of the conditional.

If we don’t do that, our test is inherently harder to write. So its slower to produce and as such a barrier to producing a unit test.

SOLID code

Part of SOLID design is to create classes that have single responsibilities. Then we create a contract for them and depend on that contract. Letting us change the behaviour of our application as long as the contract is maintained. In a unit testing scenario, this lets us insert fakes/mocks more easily.

In writing testable code its important to strike a balance point between breaking the problem up into contracts you depend on and having an explosion of classes. One code smell is a constructor with many dependencies. If you are in this situation its not always bad – but for greenfield code its certainly something you should look at and review with colleagues.

Not only does this make your unit test harder to setup it can also mean your interface contracts are not fit for consumption. At the lowest level in your API, this is fine but as you go up towards the consumer I strongly recommend that the interfaces define key application ‘operations’ rather than lots of smaller methods.

This is obviously project dependant and some projects you want a simple external interface and in others you don’t. But even so, its worth having at the back of your mind.

Summary

These are just some of my thoughts about unit testing and how it relates to software design. I’ll follow these up with some thoughts on designing your software and my take on bottom-up vs top-down design and how that affects your project scalability.

Don’t Fight The Framework – Unit Testing On Sitecore Pt I

Before I can proceed on in the Life Through A Lens series of blog posts, I thought I would cover something that has been somewhat of a bug bear of mine since I started working on the Sitecore platform, this is Unit Testing. I wanted in particular to address some of the misconceptions with unit testing and on using Glass Mapper in unit test scenarios. I have been using TDD (and as we all do Test After Development) since around 2007 and have learned a lot in this time. Due to this – I have decided to write this post before continuing with the Life Through A Lens Series to give a backdrop of what I feel is a key idea behind unit testing. I feel it is really important to cover off some of the classic TDD mistakes before I get involved with actually demonstrating how I use Glass Mapper to solve some of these problems.

I will before I go any further give a massive nod of the cap to Simon in his improvement of my knowledge in this area. I did know some things on this subject, but his patience and teaching allowed my knowledge to progress significantly beyond where it was at a rate quicker than it would have done otherwise.

Ok, so with the above in mind, I will move on to some of the most common phrases I hear from developers / managers ( / whomever ), these are usually in response to my first day questioning of ‘What unit test / mocking frameworks do you guys use?’. The answer of ‘none’ ( I don’t know why ) still offends me.

Continue reading

SwitchMasterToWeb – updated for content search & analytics

A couple of weeks ago I was tasked with setting up our UAT environment, in doing so, I duly dug out the latest copy of the Scaling Guide for the appropriate version of Sitecore, and of course, dug out the trusty SwitchMasterToWeb.config.

So, some hours later, I pushed all the necessary code for our UAT content management and content delivery servers. I was disappointed to find that our indexing was not being built on the content delivery nodes. I eventually tracked it down and found that it appears to be the index update strategy. After patching it out

My Complete File

So here is my version of the switch master to web configuration

<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/" xmlns:set="http://www.sitecore.net/xmlconfig/set/">
  <sitecore>
    <settings>
      <setting name="Analytics.DefaultDefinitionDatabase" value=“pub“ />
    </settings>
    <sites>
      <!-- OPTIONAL FOR CHANGING TO PUB DB -->
      <site name="shell" set:content="pub" />
      <site name="modules_shell" set:content="pub" />
      <site name="testing">
        <patch:delete />
      </site>
      <!-- OPTIONAL FOR CHANGING TO PUB DB -->
	  <site name="MySite">
		<patch:attribute name="database">pub</patch:attribute>
	  </site>
    </sites>
    <IDTable>
      <!-- OPTIONAL FOR CHANGING TO PUB DB -->
      <param connectionStringName="master" set:connectionStringName="pub" />
    </IDTable>
    <databases>
      <database id="master">
        <patch:delete />
      </database>
      <!-- OPTIONAL FOR CHANGING TO PUB DB -->
      <database id="web">
        <patch:delete />
      </database>	  
      <!-- STILL NOT 100% SURE THIS IS NEEDED, BUT IT WORKS FOR US -->
	  <database id="pub" singleInstance="true" type="Sitecore.Data.Database, Sitecore.Kernel">
		<Engines.HistoryEngine.Storage>
          <obj type="Sitecore.Data.$(database).$(database)HistoryStorage, Sitecore.Kernel">
            <param connectionStringName="$(id)" />
            <EntryLifeTime>30.00:00:00</EntryLifeTime>
          </obj>
        </Engines.HistoryEngine.Storage>
        <Engines.HistoryEngine.SaveDotNetCallStack>false</Engines.HistoryEngine.SaveDotNetCallStack>
      </database>	  
    </databases>
    <search>
      <configuration>
        <indexes>
          <index>
            <locations>
              <master>
                <patch:delete />
              </master>
            </locations>
          </index>
        </indexes>
      </configuration>
    </search>
    <scheduling>
      <agent type="Sitecore.Tasks.CleanupFDAObsoleteMediaData">
        <databases hint="raw:AddDatabase">
          <database name="master">
            <patch:delete />
          </database>
        </databases>
      </agent>
      <agent type="Sitecore.Tasks.DatabaseAgent">
         <patch:delete />
      </agent>
      <agent type="Sitecore.Tasks.DatabaseAgent">
         <patch:delete />
      </agent>
      <agent type="Sitecore.Tasks.DatabaseAgent" method="Run" interval="00:10:00">
        <param desc="database">core</param>
        <param desc="schedule root">/sitecore/system/tasks/schedules</param>
        <LogActivity>true</LogActivity>
      </agent>
    </scheduling>
    <contentSearch>
	  <indexUpdateStrategies>
                <!-- OPTIONAL FOR CHANGING TO PUB DB -->
		<onPublishEndAsync>
		  <param desc="database" patch:instead="param[desc='database']">pub</param>
		</onPublishEndAsync>
                <!-- THIS CAUSES INDEX ISSUES ON THE CONTENT DELIVERY NODES -->
		<syncMaster>
		  <patch:delete />
		</syncMaster>
	  </indexUpdateStrategies>
	  <configuration>
	     <indexes>
                    <!-- THIS SHOULDN'T REALLY BE ON CONTENT DELIVERY -->
			<index id="sitecore_master_index">
				<patch:delete/>
			</index>
                <!-- OPTIONAL FOR USING A PUB DB -->
			<index id="sitecore_web_index">
				<patch:delete/>
			</index>
		 </indexes>
	  </configuration>	  
	</contentSearch>
  </sitecore>
</configuration>

Please note, this version also changes the content delivery servers to use the ‘pub’ database which we have set up as a publishing target to allow the content editors to have the ability to view a pre-live version of the content by publishing to the web database.