Returning to Costs to Expect, finally!

My own projects have taken a back seat whilst I’ve been busy working on a freelance project which scaled well beyond the initial scope, that project is almost over and I suspect I’m going to be free for a little while to work on my own projects again. I can start pushing towards the soft release of the Costs to Expect API and App.

The API

The API has been feature ready for a while, its issues are unnecessary complexity. When I started adding new “item-types” I opted for a little too much abstraction. I’ve been designing the budgeting system (on paper) for a while, it is not going to work in the same way as the existing “item-types” so I’ve been busy removing all the abstractions and in general refactoring the shit out of the API.

I didn’t look at the API code for a year, that time gave me a fresh perspective, I’m simplifying the majority of the “item-type” code, there is no need for everything to extend from a base class if there is going to very little overlap in functionality.

In addition to the refactoring I’m moving our tests local, rather than relying on a Postman collection I’m moving all the tests to PHPUnit. I can replicate every test in the Postman collection locally as well as test things that could never be caught by only looking at responses. I expect I will keep the Postman collection around because there are tests that a simpler to do with it but having everything local is going to be a major benefit and of course necessary for the official release.

I have 11 stories left in my tracker, the API will then hit our soft release milestone. This milestone has been a long time coming, I still can’t quite believe we are this close.

The App

The App needs more work, there are 24 stores in my tracker, my wife and I are yet to pick the App apart, so, more stories will get added. App development tasks are simpler, the API does the heavy lifting, the majority of the work on the App is presentation.

The App is closed source, the API is open source, we handle each slightly differently, with the App we are focused on the features and not so much the overall design, assuming it works, with the API, the design takes a front seat as it has to potentially be maintainable by more than just us. The reality is, I develop everything so I’m not really switching styles, we just might approach the backlog in a different way.

Official release

Once the App and API both hit their soft release milestones, I’ll cry, no really, this has been such a long time coming, I’ve been planning the budgeting and forecasting system, forever.

There might be a slight detour whilst I develop another App rather than get on with the Budgeting and Forecasting. As mad as that sounds given how long it has taken to get here, it can be thought of as an experiment/proof of concept. The budgeting and forecasting will not behave as per anything else in the App so I need to test the API and my front-end skills.

The next four months are going to be full-on, all we can do is see where we are in September and go from there.

Jobs and tasks gamble paid off

I have spent the last 12 months developing a product for one of my long-term clients, I’m going to refer to the product as EWAQO for the rest of this post.

I initially thought EWAQO would be three to four months of work, lockdowns, homeschooling, the pandemic in general and feature creep put paid to that.

EWAQO is far from the largest product I’ve worked on, I’ve been a professional developer for over twenty years and have worked at scale, it is however the largest product I’ve created single-handled. The Costs to Expect API, Website and App are big when combined, EWAQO is in a world of its own, it has so many different systems and is immense.

I decided early on to go deep with queues and scheduled tasks, I wanted to keep the App fast for the user and do as much of the heavy lifting as possible behind the scenes. Typically, I’m using queues for complex user-initiated tasks/actions and scheduled tasks are for processing.

It is easy to handle the complex stuff behind the scenes, but there is a problem, you need some visibility. Before I wrote the first job I created the interface, that way the client can see what is happening, is a job running? Did it fail? What was the output? Who initiated it? etc.

This was the masterstroke and made so much of the complexity possible, the client could see what is happening, job classes and tasks are simpler to create because you don’t need to worry about the front-end, it all came together brilliantly.

I’m not done with EWAQO, as of writing, there are 27 jobs and 26 scheduled tasks and I have visibility of each one, I can see when they are running, what is due, when and why they fail, it all just works.

We are working toward the soft-release of Costs to Expect this year, part of that is developing two smallish apps that act as companions. Both apps will borrow heavily from what I learned developing EWAQO.

Website caching, simple now, later, dependant caches.

Two weeks ago, I quickly added caching to the Costs to Expect Website. My goal was to reduce the number of unnecessary API requests the Website made to the Costs to Expect API, mission accomplished.

The problem, I did a quick job, improvements needed.

The Costs to Expect Website is a long term social experiment, my wife and I are tracking the total cost of raising humans; we want to know how far off the £250,000 per child figure we will be when our sons reach the age of 18.

Back to the caching, issue one, I added caching to the top four or five pages; the rest of the Website needs some love. Issue two, for some unknown reason, I decided 15 mins was a sensible cache lifetime.

Solution one

I hacked in the caching; I added a simplified version of the much more featureful caching in the Costs to Expect App. I need to refactor the ‘request’ code and add caching for all requests.

Solution two

I set the cache lifetime at 15 minutes, why I don’t know. The content on the Website changes at most daily and additionally there is no need for the data to be live; people are not going to ‘throw a fit’ if they can’t see the latest expense we have added for Jack or Niall.

I am going to set the cache lifetime to four hours.

Four hours you say, why not 24? Well, I figured it is a sensible limit to ensure there isn’t too much of a mismatch between cached data while still dramatically reducing API requests.

Imagine a scenario whereby a user browses to the site and visits the summary page; the results are cached; they never, however, make it to the lists for that summary. If a second user comes along three hours later and views the listings, there is a good chance the data will mostly match the cached summary. If I set the cache lifetime at 24 hours, a value that initially seems reasonable, I am increasing the chance of the summaries and data mismatching.

There is a solution to the inconsistent data problem, dependant caches.

I need to add support for linking cached data, for example, a summary and the related lists, and more importantly, controlling the maximum period allowable between dependant cache item creation.

With the current implementation, there can be a difference of up to four hours between summary and list cache creation, realistically, the limit for dependant data should be closer to five minutes.

I will eventually update the caching system for the Costs to Expect App and at some point, trickle the implementation down to the Costs to Expect Website.

Costs to Expect App: v1.00.0

I released the alpha of the Costs to Expect App yesterday. It is later than planned, but I don’t want to dwell on that, I have other posts that explain the delay.

I am now going to work towards the public alpha. I’m not going to adjust the release date; I am still hoping to have it ready for the 1st of April, as we get closer to the release I will update the roadmap accordingly.

Rotator cuff and programming

At the start of the year, I had an accident, the result, a rotator cuff injury that is slowly healing. I still have a long way to go before my shoulder is back to normal strength and mobility.

A shoulder injury doesn’t sit well with programming, my productivity for the first three weeks after my injury dropped to zero. I’m back at work; however, I am not yet at 100%, I’m working in bursts.

I initially planned to release the private alpha of the Costs to Expect App in November. Minor delays meant I pushed it back to December. Due to injury and typically development delays, I released the alpha yesterday.

Delays and setbacks with development are common. I use Pivotal Tracker; after years of use, my iterations are typically spot-on. In my case an iteration is two weeks long, it might contain 60 points worth of work, I’ll create ‘n’ new tickets and ‘m’ bugs. By the end of the iteration, I’ve mostly completed everything I had planned.

There are weeks when I work less, and I’ll adjust the team strength value to take into account absence, but when you don’t work for three weeks, the impact is enormous. It would be easy to think I’m three weeks behind, that isn’t true, I didn’t do three weeks of development however in that period I would typically create ‘n’ new tickets and ‘m’ bugs.

I can never recover the lost time, that is gone, I’m going to use this as an opportunity to review the ‘plan’ and re-prioritise. As good as the ‘plan’ is I’m reasonably confident there are some ‘likes’ I can move further down the list and ‘needs’ that can move up the list.

Costs to Expect API v2.04.0 in development

House decorating continues, we expect to be finished sometime within the next two weeks. Decorating is limiting my development time; as such, I have decided it would be a bad idea to release v2.03.0.

If I release v2.03.0 now, it will probably be two weeks before v2.04.0 is ready. These releases are interim releases, combined they complete a significant feature. A two-week gap between the versions could cause issues so I’m releasing v2.03.0 internally and will make another public release when v2.04.0 is ready.

We aren’t due to start another round of decorating until February 2020, so typical development cadence should return in November.

v2.03.0 of the API is on the way.

The development of v2.03.0 is almost complete; I expect to have it out before the end of the week.

For this release, I am refactoring the multiple item types code. Adding support for simple expenses increased the item code significantly. New item types are due to be added to the API, so it makes sense to refactor before the complexity increases additionally.

We are decorating our house, and the scope of our plans increased, there is going to be a negative impact on my development time.

Although the pace of development has slowed, it is going to pay dividends in the future. I expect the pace to speed up once we complete decorating and I have pushed out v2.04.0 of the API.

The release is too big!

It isn’t unusual for me to release at least one new update a week for the Costs to Expect API and website. I find it significantly easier to manage and test with small iterative releases. As the only developer on this project, anything that makes my life simpler is a plus.

Occasionally a release has to touch a substantial part of the Costs to Expect API, v2.02.0 is an example of that. I need to review every section of code relating to items.

I intended to split the development work for multiple item types across two versions, v2.00.0 and v2.02.0. The database gets upgraded, and then I hit the code and expose the feature.

In planning, I appear to have overlooked summary routes. Rather than extend the design I developed to incorporate item summary routes; it makes sense to modify it slightly, given the ‘new’ requirements. If I soldiered on and upgraded the item summary routes, I suspect I would want to refactor them in a few weeks. Rather than face that situation, I feel it makes sense to go back to the drawing board and ensure I develop a design that will work going forward.

Post-release of v2.02.0, I will rework the current design, refactor as necessary and then add support for multiple item types to the item summary routes.

I gave myself one to two weeks to develop support for multiple item types. It turns out, two weeks is probably about right given the increased (actual) scope.

Three iterations of work left.

Upon checking my tracker this morning, I saw something that I haven’t seen since I began using Pivotal Tracker almost 1400 stories ago, the end of the backlog.

Am I done? No, far from, the last 14 months has all been phase one.

Unless I create additional tickets, which we both know I will*, phase two is due to begin at the end of October, the start of November.

I’ve made a point of not planning for phase two; I’m attempting to stop myself from getting distracted by future goals rather than the problems I need to solve now. I have been planning; I’ve just been trying to keep it all in my head and not put pen to paper.

In October I’ll start planning in earnest. Hopefully, the foundations I have created are suitable, I’m confident they are, but until I begin planning phase two, I will not know for sure.

*Alternatively, I could dive into the icebox and pull out a few tickets.

Costs to Expect API v2.00.0, well v2.00.1

Yesterday was release day; I released v2.00.0 of the Costs to Expect API and pushed two minor updates to the Costs to Expect Website and Web app.

Other than two minor issues, the releases went better than I expected. I did release a small hotfix that fixes one of the problems as it was publically visible, the other will get resolved in v2.01.0.

I have two Postman monitors that run daily; I also run these regularly during development to try and ensure I don’t break anything. Collection one contains all the HTTP requests the Website makes; collection two is more thorough and has HTTP requests for nearly all API endpoints. The two collections include around 1200 tests, these test the HEADERS, returned status codes and the response body. There were no tests for the issues; they slipped through the net. I have added a ticket to my tracker to add tests to catch any possible regressions.

My Postman collections are far from perfect; they give me some confidence, but as is evident based on the v2.00.1 release, there are still gaps.

I’m going to work on improving the monitors, for the last six months, anytime a feature gets added to the API, a test gets added to the relevant collection. Testing is going to be especially important when I start developing the app in a few weeks.