Continuous Delivery at SmartWeb: From a CTO perspective
created on during Continuous Delivery
Releasing a software update can trigger a wide range of emotions for a developer; Liberation, Happiness, Pride. It can also cause a lot of irritation, frustration or Rage. At SmartWeb we are very focused on releasing software updates associated with positive emotions. It should feel good (and possibly fun) – before, during and after a function transitions into production. We think Continuous Delivery is the best possible development method to achieve this specific goal.
Continuous Delivery is basically about producing and launching software faster, but there are also many other facets of Continuous Delivery. I would like to explain the three main reasons why SmartWeb is committed to Continuous Delivery.
1. We want to be faster
The development department at SmartWeb has one goal: to deliver value to our customers. Whether it is a new feature, bug fix or an internal improvement of the system. The total value can to a certain extend be calculated in the amount of releases we do. Small releases are faster to push out and produce continuous feedback together with the possibility to quickly make adjustments. Therefore, we prefer to make small but frequent releases as this produces the best possible value.
If a release is ready it has most value when it goes into production where the customer can make use of it right away. This is the same logic as when you produce cars. Cars are not simply produced and left to sit in a warehouse to gather dust, they are sent out to the sales location or to the customer who has already ordered the car before it was produced.
Speed means everything. Therefore, it is important that the development process for every release is as quick and automated as possible.
2. We want to have the best development environment
The development environment and the processes needed to develop features can be compared to the importance of a good working environment, e.g. disruptions or noises can be as equally frustrating as a release process. Here are a couple of examples:
- A developer needs to investigate a bug/issue/problem as it occurs in production, but does not have the possibility to reproduce or correct the error outside production. Bug investigating, debugging and correcting will therefore be addressed in production with risk of breaking more than just the offending bug.
- Releasing a feature is a manual process, where files are copied or merged together manually. As there is a risk of missing dependencies, forgetting files and merging incorrectly this will of course produce a small margin of error.
- Releases fail occasionally, due to wrong settings in the server environment. This means that these releases cannot be trusted and as a result sometimes only half of a release is in production, resulting in errors.
Above are examples of a development process that could be better – and become demotivating to work with for an extended length of time. Demotivated developers work both slower and make more mistakes. Therefore, it is important to improve the development environment as an integral part of the culture – both in the department and individual teams. Also that everyone is a part of making the process better.
3. Responsibility of the individual developer
Finally, it is about giving the individual developer more of the responsibility and thereby a more interesting job. They should not depend on others in the company, a large release in about 2 months or long QA process. The team has the responsibility to carry a feature from start to finish – completely into production on their own. It is a big responsibility and can be very exciting, when the development process is functioning correctly.
The QA process is a good example, as it can delay a new feature. We want the individual developer to self-assess to what extent a feature requires the QA process. The changing of a text string or color of a button does not require QA, but larger things can require: a peer reviewing process, test or people from other departments to test the feature.
We believe that at the end of the day, this responsibility results in more of everything – faster and better features in SmartWeb.
This is the background story of why we chose to use a lot of time and energy on Continuous Delivery. We have been through a long journey – and view an even longer journey ahead of us. Continue reading if you want to learn more about where we came from, where we are going and some of the challenges that we are facing right now.
Continuous delivery makes development of software faster by shortening feedback loops.
Where did we come from?
SmartWeb has always been one single version, that is updated automatically. It was a philosophy that we started with, and it is still one of the pillars of the concept that our customers purchase today.
Back then, we had one test server and one production server. There were only 2 developers on the project – if they had to correct anything, they proceeded to develop and test the feature on the test server and then manually add it to the production server. We were quick back then – it would take around 30 minutes to fix something. And the customers loved it – we could pretty much correct errors while they were on the telephone.
A lot has happened since then, and everything has become a lot larger. We have a bigger system, more developers, more servers and more customers. It requires a completely different process today, e.g. to get a bug fix in production. We have a completely different attitude toward our processes and quality than we had back then. And it is necessary, because now we have e.g. 3 developers working in the same part of the system with 3 different features. But processes and quality take time to: test, peer-review, involve others and meetings. And that is precisely why it’s important to remember the 30-minutes from back then. The challenge is to keep the speed, while everything grows.
Where are we heading?
Here are some key areas where we changed our everyday lives in relation to Continuous Delivery.
DevOps and Culture
DevOps is a blending of “development” and “operations”. It is a relatively new way of thinking for us, where we take more responsibility for the operating environment. It makes sense in many ways as good software depends on the operating environment. At the same time, the developers do not need to wait on the operating people as it can slow the process down.
This is why we officially have a “DevOps” person on the team, they are responsible for improving and maintaining the development and operating environments making sure they interact well together in the best possible way.
It is not something that a DevOps can or should bear alone. Therefore, Continuous Delivery has become part of our culture and is something that everyone on the teams should have a focus on.
Automation
We will automate everything that can be automated. It is out mantra.
Specifically, we are working toward more automation of the:
- Test Process – when we test a function. Or more importantly, test the other functionality that may be indirectly affected by a new feature.
- Release Process – when a feature is rolled out to the production servers. Or when, in the worst case out of production.
With automation comes the feeling of stability and uuniformity – that releases work when they are pushed into production. This is important to achieve speed.
Insight and Overview
We would like to have a better overview of the releases, both in the development apartment, but also the company in general. You can view all the official releases through the news list in the SmartWeb administration dashboard. But on some days we push out 10 releases, e.g. smaller optimization or bug-fixes, which is hard to get an overview of.
We want a dashboard overview of our pipeline and releases.
What are our challenges right now?
With change comes challenges. Here are some of our biggest challenges right now.
We are changing our process and culture
We are changing both our process and culture. It is a challenge that will take time to get used to, and the two things go hand in hand.
E.g. we have recently changed our deploy process, so it is more automated. It draws on the trust and culture we have around releases. But the culture takes time to change with people in the team.
At the same time, we would like to have the rest of the company with us, this is an important part of our culture, that we can deploy “all the time”. This requires a lot of trust that goes both ways.
Automated Testing
Granted – we have not been good enough for automated testing. And now it’s biting us in the ass.
We now have the challenge to introduce more automated testing into our development process. Right now, we are focused on the user-side i.e. testing of our customers stores. It is our customers lifeblood that their stores flows from Frontpage > Basket (Shopping Cart) > Checkout and Confirmation. Of course, with the correct prices etc.
It is an important part, to gain confidence and thus speed with Continuous Delivery.
Legacy
To significantly change a system and a process that operates well, can be compared to changing the tires on a car that goes 100km/hour. We should take into consideration that the system is being used all the time – we cannot simply take it offline for a significant period – this is both a challenge, but it also makes it that much more exciting.
We are always adjusting the development and operational environment which is an important part of the production environment. It takes time and is very challenging.
This was a small introduction to our journey with Continuous Delivery which is far from over. This article will be the first part of a longer series of articles where we share our experiences with Continuous Delivery.
I hope you enjoyed reading it and feel free to comment or send me an e-mall.
FAQ
Q: Why don’t you plan when you release something new and then tell us beforehand?
A: This would be the opposite of Continuous Delivery in our opinion. This would entail the planning of a release date, sharing information before and release at the exact specified time. Taken to the extreme this would be like making set release packages and release every three months, to share information about it 2 weeks before. At the release date, the release package would be much larger than what is used in Continuous Delivery. This would demand much more operational force. This process is not what we want in SmartWeb.
In some situations it can though be necessary to be one step ahead and keep the customers in the loop. Maybe the day before the release, during and after. We tried this recently with a larger restructuring of the design in our administration. That release needed more communication during the process.
Q: When do you push out releases?
A: Monday to Thursday mornings primarily. If it’s a low risk release we do push out in the afternoons and on Fridays. We can, in some situations, release at other times as well, also outside of office hours.
Q: When will we see news in the system?
A: You will see updates in the newsfeed when we release new functions or make changes which can create value for most of our customers – or the ones using the specific function.
We will not update the newsfeed when it relates to smaller bug-fixes or adjustments that are only relevant to a small number of customers. The customers who report a bug will be notified directly from the Support. We are considering a more open approach to the bug-reporting.
Q: Can I see changes that are not in the newsfeed?
A: No, not as it stands right now. Only in the part of the code which is open – our template system. Here you can see what is happening through BitBucket at https://bitbucket.org/smartwebdk.
Q: Why do you release a feature which is not finished?
A: Sometimes we get this questions from our customers. But this is exactly the point of our development process. We want to push smaller but more frequent releases. That way it’s easy for us to release a small portion of the function (which should of course work and solve a need) and then continuously keep developing the same function – if the demand is present.
Q: Is there nothing bad about Continuous Delivery?
A: We do not think so. Of course, there are downsides to every choice but we support the method fully.
