Software Development – Culture and Practice
One of the most important tasks in software development is to set up a team culture of good practice and continuous improvement, to always strive to improve the methods you use. If you don’t you can end up in a culture of chaos, bad codebases, missed deadlines, disappointment and low motivation. Fortunately best practices have been developed to help.
One of the most important tasks in software development is to set up a team culture of good practice and continuous improvement, to always strive to improve the methods you use.
Taming a bad codebase can mentally drain developers, leading to stressed and unmotivated developers which can result in a high turnover of staff. Any changes to code will take far longer resulting in missed deadlines and wasted resources.
If you don’t continuously update your development processes and software you may one day find yourself working with ancient systems, and having to do a major change instead of continuous and incremental changes. Small incremental changes are always easier to implement and far less risk prone.
The hardest part, as is with any business change is always convincing other people to follow good practice, and to use new methods. Incremental working practice and process improvements can seem minute to and insignificant to other hard pressing business tasks. The goal is to convince people that marginal gains from incremental improvements will in the end sum up to large gains over the medium to long term future. Implementing change is difficult and you can end up alienating yourself if you push too hard, and nothing happens if you don’t push at all. The right balance and approach is critical. If you don’t gain the commitment to a change and the ‘want’ to implement the change from everyone in the team, it is most likely the change will revert and fail.
The goal is to convince people that marginal gains from incremental improvements will in the end sum up to large gains over the medium to long term future
The lower down the company hierarchy you are the more skilled you have to be at relationships and influence to implement change from the bottom up. It’s much easier to implement change, when you have a lot of formal control. You don’t have to justify yourself, or persuade anyone. I personally respect anyone who wants to change/improve the processes I have set in place with their own ideas and enhancements, even if I disagree as it shows their commitment to improvement and independence. Others will feel threatened or offended and prefer conformity, you have to accurately judge people and company culture, getting it wrong can be dangerous for your relationships. It’s your risk to judge. I find the earlier you introduce good practise the easier it will be. Even if you have a small team, setting up the correct culture can pay great dividends for future growth.
Left alone software development can so easily become chaotic, and taming it can be difficult. If deadlines and expectations are missed, it can be easy for stakeholders to get frustrated and try to “put their foot down” on a project, and drown it in needless processes, ridged deadlines, numerous gantt-charts and so forth. Such ideas never work, missing an earlier task deadline can completely mess up the deadlines of dependent tasks, cascading all the way down a pipeline. This can lead to a culture of “death marches”(Often resulting in tired developers and a bad code base, slowing down all future development for a short term gain) desperately trying meet deadlines and stressed developers, again draining them and turning attitudes negative making the problem worse. Getting the right balance is crucial! It’s in the nature of software development to resist such attempts, since you can never accurately predict how long something is going to take from developer/analyst estimates. The methods in the post will help, and if you follow the methodology your estimates(which will be partially be based on previous data, rather than developer/analyst guesses) will continually improve and be refined.
I personally advocate lightweight development methodologies, such as Scrum and Kanban(Which is more about making work, and processes visible then a complete methodology. However making things visible often forces you to formalise things a bit more) depending on the problem. I combine these methodologies with Continuous Integration(CI), Automated/Continuous Deployment(CD), Automatic Unit Testing and Integration tests, TDD(If appropriate), avoiding long-term source code branching and automating as much ‘process/pointless’ work as possible. Continuous Integration, Continuous Deployment, and a lot of other methods in this post are all about refinement and marginal gains which add up in the long run. Small gains, that can overall can add up to be powerful solution. The problem is in isolation they can seem insignificant, but together can make or break a project.
Continuous Integration servers constantly scans your source repository, and once it detects any changes starts to build a working version of the software. It can integrate latest version of libraries and external projects into your current project and outputs a working build of your software. This verifies that your software constantly works and interfaces with any external projects. Continuous Integration also saves build time, and your developers will always have a latest working copy for any stakeholders to inspect. If your build fails, you know instantly and can react accordingly. Anything that can be scripted, and automated which your build requires should be included in the continuous integration cycle, this removes any ‘process’ work for developers allowing them concentrate on getting the product right, and makes things consistent and predicable for each build. Continuous Integration makes turn around time for each change or feature much faster thus allowing for almost instant feedback.
Automated Testing – Continuous Integration(CI)
CI also greatly benefits automated testing of software and thus the quality of your software. On every build it should run your unit tests, integration tests, user interface tests, and any other automated tests. If your tests fail, your developers will know instantly allowing them to fix the problem quickly instead of having to wait for a manual build, and accidentally building on top of broken code! Previously it could be many hours, or days before a complete build was manually invoked, so problems can be hidden for some time.
Release Management – Continuous Integration(CI)
The majority of CI(Continuous Integration) systems also have release management facilities, allowing you to ‘promote’ a build for a public release, and then storing those builds forever and tagging it in your source repository. Any customer complaints against a specific build can then always be re-visited, or builds reverted to old versions in case of disaster.
Branching - Continuous Integration(CI)
In general and especially when working with CI I recommend avoiding long-term source code branches if possible. Branching and merging can be major time sinks, and can introduce major bugs and nuances if not done correctly. This is especially true if branches have massively diverged away from each other, merging can become almost impossible. When using CI make sure you always have working mainline copy(Don’t commit bad code), with everything included. This should always make sure everything is tested, and consistantly ready to be deployed.
If you have features, changes that can not completed in one go you should first try to design your software so that things can be implemented incrementally, with no user interface options to use that feature. If you can not do that, use feature toggles in your code and only as a last resort use branches.
Short-Term(Hours) branches are acceptable if they used in a code review system or similar, such as making sure code commits to certain standards before merging into the mainline. Obviously automate your code review system. When your branches only last hours, it is likely that the mainline and branch have hardly diverged.
Quick Take Away Points
- CI continuously verifies your software is working, and continuously generates the latest working build for stakeholders.
- Use CI to automate as much ‘busy’ work as possible, allowing developers to focus on the product.
- CI continuously verifies the quality of your product, allowing for faster feedback if something breaks
- Make sure your code mainline is always in working condition, and avoid branches.
Tools And Quick Links
- Jenkins – A common continuous integration server.
- Hudson - A common continuous integration server.
- Bamboo – A good commercial continuous integration server.
- TeamCity - A good commercial continuous integration server.
- Travis CI - A CI server popular in the open source world.
Continuous Delivery is about automating deployment, and releases of code. The main principle is that committing code can automatically invoke a process of building, and deploying to a staging environment or an executable release without any developer intervention. In order to use continuous delivery, you need to be using continuous integration as it is a required dependency as it always makes sure your software is in usable condition. CD is usually implemented as scripts which are automatically invoked by a CI server.
Automatically deploying code can save time, and remove any mistakes in manually deploying software. It essentially standardises the deployment of code, making it predictable, repeatable and bug free. Deployment of code using this process becomes a non-issue for a company, where as before it felt like a stressful time. Your deployment scripts can automatically set up infrastructure like, load balancers, servers, databases, and messaging queues from a public cloud by defining infrastructure as code using a infrastructure language such puppet, chef or scripting against the cloud providers API. Once infrastructure is set up correctly deployment of code/executables can commence. Once you are comfortable with that particular build after testing it in a staging environment you can then ‘promote’ a build to production. This essentially repeats the exactly same process but against a production environment, and permanently tags the release in your source control so you can always refer back to it. You should be able to promote almost every build from staging to production, if your unit tests, system integration tests, and other processes are working correctly otherwise something needs to change. It makes the process of releasing software fast, easy, predictable, repeatable and bug free, and because of this it is one of my favourite practices to implement and removes a lot of hassle.
Frequent Releases - Continuous Delivery(CD)
Continuous Delivery also promotes a culture of frequent small releases to production/customers rather than single large releases. Often developers are used to releases which last several months, or years. Continuous Deployment can reduce this days/weeks, which for certain environments is a very good thing. It is what makes CD a good fit for software as a service applications, where applications are often hosted by the provider rather than customers.
A big problem with large single releases is sudden change, which can end with shock, disaster and time wasted in developing features customers never really wanted. Using a frequent release cycle these problems can be alleviated. Frequent releases also allow for fast feedback from customers/users, allowing the market to continuously validate the direction of your product. Customers and user are often far more comfortable in adapting to frequent small changes, rather than a sudden large change that forces your customers to suddenly change their processes which interact your product.
Continuous Deployment - Continuous Delivery(CD)
Continuous Deployment takes continuous delivery to the next level. Normally you have some kind of manual intervention, between deploying to staging and deploying to production. This often comes in the form a “promote to production” button or similar which needs to be pressed in order to deploy to production. Continuous Deployment removes the manual intervention preventing deployment straight to production. Most people have to feel extremely confident in their development processes to allow for continuous deployment but people do use it.
Continuous Delivery(CD) in Mobile Development
People developing mobile applications often think the way the app store works conflicts with Continuous Delivery. However there is nothing stopping developers from automatically pushing to your beta testers, using TestFlight/HockeyApp and other such services for fast feedback on your product.
Quick Take Away Points
- Automate Deployment, and constantly deploy to a staging environment similar to production using CI.
- You can even automate setting up infrastructure(Message Queues, Databases, Servers, ESBs) using tools like puppet and chef.
- Continuous Delivery reduces the risk of any deployment issues, and makes deployment a non-issue
- There’s no excuse not to use CD in mobile development
- If you can try go for fast release cycles, allowing for quicker and better feedback from customers.
Tools And Quick Links
Continuous Inspection is set of tools that continuously monitor(Often by being integrated in your CI Server) your code base, and reports any quality metrics on the source code. These metrics allow you to keep on top of complexity in your code base before they become too much of a problem to handle. They can provide a quick but basic overview of any problems in your source code. You can track these metrics as time goes on, and see where complexity and problems are growing and what might need to be refactored. Most continuous inspection software comes with a dashboard(Separate Portal or Integrated into CI Server) which allows developers to quickly check up on problems developing in the code base which need to be refactored.
NCSS - Non commenting source statements. The amount of statements in a particular module or function. If it becomes to big, it probably means your function is trying to do too much and should be split into multiple functions or methods.
Cyclomatic Complexity – It’s metric that counts the number of ‘decisions’ or ‘branches’ such as if statements in a particular module. Its best keep this approximately below 10 in a method before refactoring. There is no hard and fast rule, and its best to use judgement. When a developer writing software, they need dry run(think through) the code in their heads. Having to many decisions in piece of code means you can easily get lost, so functions with to much complexity can be bug prone. High Cyclomatic Complexity is also highly related to low cohesion(Logically organised code). Cyclomatic Complexity will never disappear by splitting functions or classes, you are just shifting it about in the program. However you are aiming for lower cyclomatic complexity for each method/function and class. For example splitting a method with complexity of 10, could result in two methods with complexity of 5 each.
NPath – Similar to Cyclomatic Complexity, but differs as it takes into account the nesting of conditional statements and multi-part boolean expressions. Normally you start refactoring methods as it goes over 100 or 200 on this metric. Again however, no hard and fast rule. You need human judgement.
Code Size of Classes/Number of Methods in Classes – This size of a class. If class gets to large, its a good sign that it is taking on far too much responsibility, and should be split into multiple classes. Classes should do one thing, well.
Coupling – The number of classes, which a class interacts with. In ideal designs classes should interact with as few classes as possible through well defined interfaces. Lots of coupling can result in a cob-web mess of interactions between classes, and headaches for developers.
There are many metrics which are available to use, and many other metrics which are composites of other metrics. Choosing which metrics to use requires an element of judgement.
Code Metrics – A Word of Caution.
Code Metrics are good as warning signs that code might be becoming unmaintainable. However metrics should not followed religiously as they can cause unintended side effects. Trying to lower cyclomatic complexity might result in functions that have been split illogically for example.
Code Metrics can’t judge higher level architecture, and things like cohesion(Although some metrics correlate with it). If your metrics are good, it doesn’t necessarily mean you have a good design.
Quick Take Away Points
- Code Metrics can we be used as warning signs to which code might be becoming unmaintainable, so you can keep on top of your code base.
- Don’t religiously follow metrics as they can cause unindetended side effects. Use them as warning signs.
- Metrics can’t judge high level design choices.
Tools and Quick Links
Sonarqube - Continuous Inspection software with its own management portal/dashboard for developers.
There are many software development methodologies, but the one I use is Scrum. When trying to approach your development methodology you have to take it seriously, you can’t half-way it, otherwise you might as well not do it all. If your not careful lightweight methodologies can easily turn into chaos pretending to be controlled. Scrum provides you with metrics(To measure yourself against), visibility(So everyone knows what’s going on), and direction(With limited resources, do we implement this feature or that feature and what are the trade off’s.) without drowning you in process and arbitrary deadlines(Developer Guesses) which have no basis in reality. One of the core principles of scrum is continuous improvement, to continuously get better at things overtime. It does this by feeding previous information(How long did that take? What slowed us down?) into future planning.
The basics of scrum involve backlog, sprints, daily scrums, planning sessions, backlog grooming sessions, sprint retrospectives and sprint completion meetings. The backlog is a list of user stories/tasks which define specific pieces of work in order to complete or improve a product. A sprint is a locked time period, such as one week or two weeks in which to implement tasks that have been selected from the backlog. Before you embark on a sprint, you have choose you the tasks you want to complete for that sprint and once selected the sprint is locked for the time period. The sprint can’t be changed half way through, or change direction, If a sprint goes wrong you start again. A daily scrum is daily morning meeting between developers within which developers discuss what they are working on, what they are planning to do, and if there are any blockers preventing them doing their work. Sprint planning sessions are meetings before a sprint, in which developers, and product owners decide what to implement. You prioritise tasks, make trade off’s between tasks and fill up a sprint a reasonable amount of work to complete. Sprint retrospectives are after sprint completion, they allow developers to discuss what went wrong in a sprint such why certain tasks took longer then expected. This information can feed into the next sprint, so your next spring will be more accurate, and hopefully more work will get done. Sprint completion meetings allow developers to demonstrate their work to stakeholders, to check if the product is on the right direction. Backlog grooming sessions involve going through the backlog, and re-writing ambitious tasks, deleting any backlog tasks which are no longer relevant, without backlog grooming sessions backlogs can grow unmanageable.
Scrum and most lightweight methodologies ensure visibility in a project, and that is of the main advantages in following a good methodology.. In scrum each task/user story goes through a workflow, such as “To Do” -> “Implementation” -> “Finished”, this workflow is completely customisable and you might want to add steps such as ‘testing’. It is common to have whiteboard, with a column for each step in the workflow, and each developer will put note on the board where they are on a certain task. Overtime the tasks will move from left to right, until all tasks are completed for sprint. Having this information visible allows stakeholders/developers to have quick overview on a project, such as which user stories are completed and what needs to be done. It is useful to have a virtual whiteboard, using web software such as JIRA to allow for distributed teams.
Scrum also ensures visibility in planning as it makes sure all trade off’s are known. Typically in software projects, there scope creep where little things are added to the to feature list until the amount of work to be completed in a certain time becomes impossible. Scrum enforces time locked sprints which only reasonable amount work for that time period can be added. If someone wants a certain feature, it makes it known that something else has give way to implement that feature. It makes these trade off’s visible, instead of continually expanding scope creep.
One of the main problems with software projects is estimation, and it often involves arbitrary estimates from developers when a feature will be completed. Release deadlines are often generated from these estimates, and when the estimates prove to be off, it pushes out all deadlines in the development pipeline. Scrum avoids arbitrary deadlines, and uses previous data to help estimate when things will be completed. In scrum developers are asked to estimate the complexity of tasks, and you can estimate in complexity points, story points, time or any other unit which denotes how hard something is to complete. The unit of complexity just has to be consistent between tasks for it to be accurate. You guess how much complexity you can complete in a sprint, and set that amount of work in the sprint planning session. When you are running the sprint, you record how much complexity you can complete on average in one day. This your velocity, and can be used to estimate time frames for releases. If for example on average you can complete 10 complexity points, and you have 50 complexity points in your backlog, you can give a good estimate of 5 days for completion. The velocity is an average of previous completion rates, and thus uses previous data to help estimate future tasks, so as you complete more tasks it get more accurate. Because the velocity is an average, estimates for individual tasks maybe under, or over but overall they cancel each other out. This roots estimates in previous completion rates, as opposed to guesses from developers. Using the velocity you can work how much work you can fit into future sprints, and you will get better at this over time.
The graph(Generated from JIRA) below shows the amount storypoints to be completed for a release, the amount of storypoints completed, and a trend based off velocity. Where the trend line exits the total story points box is the estimated released date for that particular release.
In order to keep track of the data, sprints, tasks and completed complexity points it is common to use software manage all this. I personally use JIRA to manage my projects, developers can drag tasks they’re completing through workflow on a virtual whiteboard. The software will automatically calculate velocity, and give estimates on release dates. They also provide better ways to manage tasks and the backlog, with ways to label and organise tasks. You can specify what components in the software do certain usercases involve for example. When you come to plan your next sprint in the software, you use your previous velocity to work how much work you can put in the next sprint, and overtime this will allow you to accurately predict how much work you can do in a sprint .
You can use these metrics generated by the software to measure yourself against, to see where you can improve. You can see how fast you’re burning down complexity points, if your estimating release date is slipping and so on. Most software allows you put these metrics, and virtual whiteboards on a dashboard and display it on a large screen. This allows everyone to know what’s going on in the project, who’s working on what, and if we’re on track.