Podcast - What should all software developers want in their applications?

Each week, my good friend Zach Mayer and I answer geek culture's most superfluous questions on the QQ-Cast. This past week we turned to the subject of software development, and did a retrospective on my series of blog posts, Three Things that all Applications MUST Have.

QQ Cast - Quest 40 - What should all software developers want in their applications?

• 00:00 - Mic Check and Introductions
• 03:55 - Why isn't your application logging?
• 06:15 - Where is your configuration coming from?
• 11:00 - Can we talk about Unit Tests for 18 minutes?
• 29:00 - Is continuous integration necessary?
• 30:30 - Does automated deployment speed up development?
• 38:00 - When should you use inversion of control?
• 49:30 - How intrusive is error reporting for end users?
• 59:00 - Do you remove user impersonation from production builds?
• 63:30 - Why is continuous deployment so controversial?
• 72:15 - Wrap up!

Enjoy,
Tom

Three Things All Applications SHOULD WANT to Have

This is the third in a three part series:

1. Three Things that all Applications MUST Have
2. Three Things that all Applications SHOULD Have
3. Three Things that all Applications SHOULD WANT to Have

Somehow these three things are quite controversial. I have had many debates with people who are not using these practices and services, and they are not convinced of their usefulness. However everyone I have met that has used these tools is always a staunch defender of their value! I beg you to give these a chance, try them out and they will prove their merit to you!

Do you agree or disagree with these practices? Let me know in the comments!

1. Error Reporting

How do you know what is wrong with your application? Without proof you are just guessing!

Reporting errors to a central location helps you solve problems the moment they begin, not after they have negatively impacted your entire user base. Here is a fun Microsoft statistic: 80% of customer issues can be solved by fixing 20% of the top-reported bugs. So start reporting your exceptions today!

• Exceptionless
• Elmah
• Raygun

2. User Impersonation

The easiest way to recreate a bug reported from a specific user is to actually be that user. By adding user impersonation to your application you can save your QA team hours of time. While I completely understand the security concerns of this feature, I must still emphasize the value that it returns in the forms of testing and debugging.

You may want to take this code out in production, but make sure you have in QA and Dev!

3. Continuous Deployment

There is a difference between continuous delivery and continuous deployment, and I am talking about continuous deployment. Just imagine: instant bug fixes, constant streams of new features, and best of all no more deployment schedules! This is not a cheap goal to achieve and it requires continuous maintenance of your tests and deployments, but so does all software development!

To be honest I have never worked in an environment where we made it all the way to continuous deployment, but I would really like to someday! The few people I have met who have been able to accomplish this task had nothing but great things to say about it. Like always I would suggest starting small with a new practice like this, pick an internal application or minor project and begin your foray into continuous deployment from there.

Miss a post in this series? Start over with part 1: Three Things that all Applications MUST Have

Enjoy,
Tom

Three Things that all Applications SHOULD Have

This is the second in a three part series:

1. Three Things that all Applications MUST Have
2. Three Things that all Applications SHOULD Have
3. Three Things that all Applications SHOULD WANT to Have

These three things are all generally agreed upon as being best practice, and few people will argue against their value. Unfortunately not all teams take the time to set them up. Again, I can not emphasize how much time these services will save you time in the long run, the earlier you set them up the more value they will return to you!

1. Dynamic Configuration

What the hell is "dynamic" configuration? It's configuration that is simply not not static or hard coded. Do not use constant strings or compiler symbols to configure your application! Start by using configuration files and build transforms. If your system is very distributed, consider using remote or discovered configuration.

• SlowCheetah - XML Transforms
• Scott Hanselman Blog Post

2. Continuous Integration

How do you know that the code in your source control is in a functional state? If not, who broke the build? Continuous integration is the practice of consistently pulling and building the projects in source control in an automated fashion. Typically these builds will also execute any tests associated with the project and provide reports for each build. This is crucial rapid development, and is a necessary first step on the road to continuous deployment.

• Set Up Continuous Integration with Team Foundation Server
• JetBrains Team City
• Jenkins CI

3. Automated Deployment

Before we can get to continuous deployment we have to start with automated deployment. This is simply the act of having a service that deploys your applications to an environment without the need for any significant human interaction; i.e. you can deploy with the click of a button! Automated deployment is extremely useful because it drastically speeds up deployments, prevents human error, and restricts access to different environments (such as production). Please, do not under estimate the value that a deployment system can provide!

• Octopus Deploy
• Red Gate Deployment Manager

Continue reading part 3: Three Things that all Applications SHOULD WANT to Have

Enjoy,
Tom

Three Things that all Applications MUST Have

This is the first in a three part series:

1. Three Things that all Applications MUST Have
2. Three Things that all Applications SHOULD Have
3. Three Things that all Applications SHOULD WANT to Have

I feel very strongly that when you start a new project you should spend your first day or two just setting up a few basic utilities. For every hour you spend at the beginning of a project setting up these tools you will save yourself days down the line.

1. Logging

What is your application doing? How can you debug it? Will that work in all environments? The go to answer for these questions should always be logging!

I am constantly amazed at how many applications do not have a logger. To be fair, most of the time when I do not see a logger it is because the application is small or started out as a one off project. However to me that is all the more reason to just take the time and setup a logger right from a project's inception, then you know it will always be there. Thick client, thin client, or back end service, it should have a logger!

• log4net
• NLog
• Common.Logging

2. Dependency Injection

Dependency Injection is a pattern that drives a lot of best practices: it allows you to loosely couple your modules, forces you to consider the number dependencies any given module requires, and perhaps most importantly it makes your code very testable. The inversion of control that dependency injection provides also enables you to refactor and test in ways that are almost unachievable without it.

It can take a little while to fully understand dependency injection, especially the intricacies of lifetime management, but once you understand the fundamentals you can apply that knowledge to any language and any framework.

• Unity
• Ninject
• Castle.Windsor

3. A Test Project

There is no reason not to have a test project as part of your solution. Regardless of how you feel about Test Driven Development (TDD) as a best practice we should all be able to agree that unit testing does provide value and is a good thing.

I encourage you to start simple: just create a test project with ONE unit test inside of it. Even if you do not have time to write tests right now, just having the project already setup will enable you to write them later. Additionally, just thinking about writing unit tests encourages you to author more testable code; so if absolutely nothing else then just use your test project as a best practices placebo!

• xUnit.net
• NUnit
• MSTest

Continue reading part 2: Three Things that all Applications SHOULD Have

Enjoy,
Tom

How to Test RavenDB Indexes

What if you could spin up an entire database in memory for every unit test? You can!

RavenDB offers an EmbeddableDocumentStore NuGet Package that allows you to create a complete in memory instance of RavenDB. This makes writing integration tests for your custom Indexes extremely easy.

The Hibernating Rhinos team makes full use of this feature by including a full suite of unit tests in their RavenDB solution. They even encourage people to submit pull requests to their GitHub repository so that they can pull those tests directly into their source. This is a BRILLIANT integration of all these technologies to both encourage testing and provide an extremely stable product.

So then, how do you test your RavenDB indexes? Great question; let's get into the code!

1. Define your Document

public class Doc

{

    public int InternalId { get; set; }

    public string Text { get; set; }

}

2. Define your Index

public class Index : AbstractIndexCreationTask<Doc>

{

    public Index()

    {

        Map = docs => from doc in docs

                      select new

                      {

                          doc.InternalId,

                          doc.Text

                      };

        Analyzers.Add(d => d.Text, "Raven.Extensions.AlphanumericAnalyzer");

    }

}

3. Create your EmbeddableDocumentStore
4. Insert your Index and Documents

In this example I am creating an abstract base class for my unit tests. The GetDocumentStore method provides an EmbeddableDocumentStore that comes pre-initialized with the default RavenDocumentsByEntityName index, your custom index, and a complete set of documents that have already been inserted. The Documents come from an abstract Documents property, which we will see implemented below in step 5.

protected abstract ICollection<TDoc> Documents { get; }

private EmbeddableDocumentStore NewDocumentStore()

{

    var documentStore = new EmbeddableDocumentStore

    {

        Configuration =

        {

            RunInUnreliableYetFastModeThatIsNotSuitableForProduction = true,

            RunInMemory = true

        }

    };

    documentStore.Initialize();

    // Create Default Index

    var defaultIndex = new RavenDocumentsByEntityName();

    defaultIndex.Execute(documentStore);

    // Create Custom Index

    var customIndex = new TIndex();

    customIndex.Execute(documentStore);

    // Insert Documents from Abstract Property

    using (var bulkInsert = documentStore.BulkInsert())

        foreach (var document in Documents)

            bulkInsert.Store(document);

    return documentStore;

}

5. Write your Tests

These tests are testing a custom Alphanumeric analyzer. They will take in a series of lucene queries and assert that they match the correct internal Ids. These documents are being defined by our abstract Documents property from Step 4.

NOTE: Do not forget to include the WaitForNonStaleResults method on your queries, as your index may not be done building the first time you run your tests.

[Theory]

[InlineData(@"Text:Hello",              new[] {0})]

[InlineData(@"Text:file_name",          new[] {2})]

[InlineData(@"Text:name*",              new[] {2, 3})]

[InlineData(@"Text:my AND Text:txt",    new[] {2, 3})]

public void Query(string query, int[] expectedIds)

{

    int[] actualIds;

    using (var documentStore = NewDocumentStore())

    using (var session = documentStore.OpenSession())

    {

        actualIds = session.Advanced

            .LuceneQuery<Doc>("Index")

            .Where(query)

            .SelectFields<int>("InternalId")

            .WaitForNonStaleResults()

            .ToArray();

    }

    Assert.Equal(expectedIds, actualIds);

}

protected override ICollection<Doc> Documents

{

    get

    {

        return new[]

            {

                "Hello, world!",

                "Goodnight...moon?",

                "my_file_name_01.txt",

                "my_file_name01.txt"

            }

            .Select((t, i) => new Doc

            {

                InternalId = i,

                Text = t

            })

            .ToArray();

    }

}

Enjoy,
Tom

Object Oriented JavaScript Tutorial

Over the past month I spent a lot of time helping teach a good friend of mine how to write advanced JavaScript. Although he had worked with JavaScript many times before, he did not know a lot of the simple things that make JavaScript the crazy dynamic rich development experience that it is. Together we built up this series of examples to help teach everything from language basics to object orientation.

Why do we need yet another JavaScript tutorial?

JavaScript is an interesting language. It is loosely typed, the object orientation has been hacked in over time, and there are at least 10 ways to do anything. The most important thing you can do when writing JavaScript is to choose a set of conventions and stick with them. Unfortunately with so many different ways to do things out there, it's hard to find tutorials that are consistent in their conventions.

Just like developing in any other programming language, understanding the fundamentals is key to building up an advanced application. This tutorial starts out extremely simple, and incrementally gets more advanced. It shows alternative implementations for different tasks, and tries to explain why I prefer the ones that I do. Meanwhile it always follows a consistent set of conventions.

Let's get to the tutorial!

This tutorial comes in 16 lessons. They are designed to be debugged with Firebug for Firefox. If you don't want to actually run the scripts then you can always just read them by downloading the small htm files below and opening them in notepad (or whichever text editor you prefer).

1. JavaScript Types
2. Object Properties
3. Object Declarations
4. Objects vs Arrays
5. Object Pointers and Cloning
6. Equals Operators
7. Closures
8. Advanced Closures
9. Defining Classes with Closures
10. Defining Classes with Prototype
11. Function Scope
12. Creating Delegates
13. Class Inheritance
14. Advanced Class Inheritance
15. More Advanced Class Inheritance
16. Extending jQuery

Additional Resources

If all that wasn't enough or it just got you thirsty for more, here are some additional resources to help start you down the path of becoming a ninja with JavaScript. My recommendation is that whenever you are writing JavaScript you should have w3schools open in a second browser window at all times.

• Downloads
  • Download this tutorial in a ZIP
• Learning Resources
  • w3schools - JavaScript and HTML DOM Reference
  • krook - JavaScript DOM
• Alternative Tutorials
  • Object Oriented Programming in JavaScript (Using Prototype) - Mike Koss
  • JavaScipt Closures for Dummies - Morris Johns
  • JavaScript - Crockford
• JavaScript Frameworks
  • jQuery
  • Sencha - Ext JS

Enjoy,
Tom

9/21 Update - Thanks to Zach Mayer (of System-Exception.com) for providing a great critique of my tutorial. He pointed out several typos and small bugs in lessons 4, 6, 7, 9, 10, 12, 14, 15.

Why best practices?

I recently engaged in a fun discussion: Why should we (developers) follow best practices?

My first knee jerk reaction was to say "because they are best practices!" Of course someone immediately pointed out to me what a terrible argument that was. To their point, I could call anything a best practice, but that does not necessarily make it a good thing.

Best Practices: All the cool kids (wearing fedoras) are doing it.

So yes, I agree that any particular best practice should be able to stand on it's own merit. Of course this immediately brings up the next problem...

How do we train people to follow these best practices without having to prove out each and every one?

This is where we have to put our egos aside and be willing to defer to other people's opinions. When a Microsoft MVP or your team lead comes to you and says that you should do something a particular way, you probably want to consider what they have to say. This is because they have credentials, and that they have (hopefully) earned those titles and merit badges from years of experience, which would certainly imply that their opinion has value.

Am I saying that you should just defer to your senior developers / elders? No, absolutely not! Honesty goes both ways, so if something is wrong then you call them out on it! However, you should not feel the need to scrutinize every practice someone else puts forward just because you have not used it before. Again, if something is a good idea it often becomes self evident once implemented.

If you are trying to promote a best practice, start by providing a simple example or two.

Why should we have a data access layer where all of our database code goes? There are several reasons. When a change gets made to the database (and changes WILL get made to the database), there is only one place where you have to go to update your data access logic. Putting all of your logic in one place ensures that people will not rewrite the same queries over and over again. The list goes on.

If you are the trainee, please do not fall back on using arguments from ignorance.

If you are a web developer and you are asking why should I use interfaces in your MVC models, then you probably have not worked on a large project where multiple controllers (each with different models) are all rendering the same user controls. Admittedly it may take a little bit more code to implement an interface, but the reuse that it provides more than makes up for that, and when we look at this in action the advantages should become self evident.

On a related note, why do so many developers not believe in training?

Even after making it big, professional sports players still train everyday. All of the best athletes in the world still have coaches.  The world's leading scientists still read new papers published by students. So what makes software engineers any different?

I encourage all developers to go out and read blogs, check out sample projects, research design patterns, attend their local user groups, and do anything to get exposed to new ideas! Remember that such things will often provide an immediate return in your every day work life. Testable code makes hunting bugs easier, it makes your integration phase shorter, it makes releases go smoother, and that makes your stress level go down!

Programming is fun, and adhering to best practices only helps make it more fun for everyone!

~Tom

Gathering Requirements for Open Source Projects

This past weekend was Dallas TechFest 2011. It was a great conference filled with great speakers and great attendees. Despite speaking at 9am both mornings (which I consider to be far too early in the morning for...well, anything), I had amazing turn outs for both presentations. In short, I had a blast, I got to hang out with great people, and I even learned at thing or two. But I digress.

For the abridged version, just stick to the big italic text. ;)

The Setup

Having so many community leaders in one place at one time, on Friday night a Dot Net User Group leader meetup was held, and this meeting took the form of a fishbowl. Many questions were raised, most got answers, but of course others only wound up raising more questions. Something particularly fun that happened when Devlin Liles raised the question, and I'm paraphrasing: "Why are so many SIG (special interest group) websites so crappy?"

After we all kicked this around for a while, and all the arguments seemed to boil down to the fact that everyone uses different systems to organize their groups, and there does not exists a univerially accepted and easy process to integrate them. As the conversation wound down, I hopped up in the one of the chairs and offered to help spearhead a project to write some type of new website that could help solve these problems.

Then something really really cool happened: a show of hands was called, and almost everyone in the room expressed an interest in contributing to such an open source website!

We very quickly decided to hold an open requirements gathering session the next day at Dallas TechFest (coincidentally to be held in the CodeSmith Hack Room). This is just another example of why I love being a member of the .NET community. It shows just how fun, interactive, and giving the this particular geek can be.

The Problem

Our story continues Saturday morning in the CodeSmith Hack room. Almost three dozen devs showed up to participate, meaning that our numbers had already grown since the previous night! We started out by making two lists on the rooms white board, "this it absolutely must do" and "things it absolutely must not do". The lists were drawn up, votes were cast, and then the drill down debate began.

It took us an additional 45 minutes to realize that the mistake had already been made: We had all listed off similar requirements, but we all had different visions of what this website would actually do.

It took us a while, but we finally figured out that we were all using the same adjectives and verbs to describe different nouns. Finally we called a vote to discuss if we were talking about 1) a top down approach where the website collects and aggrigates data from other SIG sites, or 2) a bottom up approach where this is just an open source SIG website that anyone can host themselves.

So, why did it take us so long to identify the problem? Also, how could this miscommunication have been avoided?

The Solution

Countless statistics and case studies have pointed to the requirements phase as the key to determining whether not a project will succeed or fail. I view this as a modest reminder that no project is immune to this, especially not open source ones. I fear that developers, especially the rock stars who like to engage in open source projects, often forget this.

In my opinion, our particular situation could have been avoid if we had started by defining some basic use cases.

We started out by focusing on functional requirements, but depending on which permutation of these that we selected we would come out with very different products. If we had started out by defining some use cases, that might have helped us determine how our users would interact with this website and thus would have helped us better understand our functional requirements.

Examples

I am not saying that we should have broken out our UML Diagram tools. I am saying that by simply defining basic use cases we can help create a context in which to define functional requirements.

Here are some examples to help clarify this.

Objective: Create a website to help people find user groups.
Requirement 1: Must include user group information.
Requirement 2: Make events appear on a searchable calendar.
Requirement 3: Search for events by metroplex or geographical area.
Ambiguity: Is this a portal to other websites, or is it a hosting service for other websites?

Now, let's add two use cases (one for the end user, and one for the administrator) to clarify things.

Objective: Create a website to help people find user groups.
Requirement 1: Must include user group information.
Requirement 2: Make events appear on a searchable calendar.
Requirement 3: Search for events by metroplex or geographical area.
Use Case 1: Users may arrive at the website and see a list of upcoming events in their area.
Use Case 2: Administrators should be able to register their existing website with this portal.
Ambiguity: Not so much.

Anyway, I hope that helps! 

~Tom