Website caching, simple now, later, dependant caches.

Two weeks ago, I quickly added caching to the Costs to Expect Website. My goal was to reduce the number of unnecessary API requests the Website made to the Costs to Expect API, mission accomplished.

The problem, I did a quick job, improvements needed.

The Costs to Expect Website is a long term social experiment, my wife and I are tracking the total cost of raising humans; we want to know how far off the £250,000 per child figure we will be when our sons reach the age of 18.

Back to the caching, issue one, I added caching to the top four or five pages; the rest of the Website needs some love. Issue two, for some unknown reason, I decided 15 mins was a sensible cache lifetime.

Solution one

I hacked in the caching; I added a simplified version of the much more featureful caching in the Costs to Expect App. I need to refactor the ‘request’ code and add caching for all requests.

Solution two

I set the cache lifetime at 15 minutes, why I don’t know. The content on the Website changes at most daily and additionally there is no need for the data to be live; people are not going to ‘throw a fit’ if they can’t see the latest expense we have added for Jack or Niall.

I am going to set the cache lifetime to four hours.

Four hours you say, why not 24? Well, I figured it is a sensible limit to ensure there isn’t too much of a mismatch between cached data while still dramatically reducing API requests.

Imagine a scenario whereby a user browses to the site and visits the summary page; the results are cached; they never, however, make it to the lists for that summary. If a second user comes along three hours later and views the listings, there is a good chance the data will mostly match the cached summary. If I set the cache lifetime at 24 hours, a value that initially seems reasonable, I am increasing the chance of the summaries and data mismatching.

There is a solution to the inconsistent data problem, dependant caches.

I need to add support for linking cached data, for example, a summary and the related lists, and more importantly, controlling the maximum period allowable between dependant cache item creation.

With the current implementation, there can be a difference of up to four hours between summary and list cache creation, realistically, the limit for dependant data should be closer to five minutes.

I will eventually update the caching system for the Costs to Expect App and at some point, trickle the implementation down to the Costs to Expect Website.

Costs to Expect App: v1.00.0

I released the alpha of the Costs to Expect App yesterday. It is later than planned, but I don’t want to dwell on that, I have other posts that explain the delay.

I am now going to work towards the public alpha. I’m not going to adjust the release date; I am still hoping to have it ready for the 1st of April, as we get closer to the release I will update the roadmap accordingly.

Rotator cuff and programming

At the start of the year, I had an accident, the result, a rotator cuff injury that is slowly healing. I still have a long way to go before my shoulder is back to normal strength and mobility.

A shoulder injury doesn’t sit well with programming, my productivity for the first three weeks after my injury dropped to zero. I’m back at work; however, I am not yet at 100%, I’m working in bursts.

I initially planned to release the private alpha of the Costs to Expect App in November. Minor delays meant I pushed it back to December. Due to injury and typically development delays, I released the alpha yesterday.

Delays and setbacks with development are common. I use Pivotal Tracker; after years of use, my iterations are typically spot-on. In my case an iteration is two weeks long, it might contain 60 points worth of work, I’ll create ‘n’ new tickets and ‘m’ bugs. By the end of the iteration, I’ve mostly completed everything I had planned.

There are weeks when I work less, and I’ll adjust the team strength value to take into account absence, but when you don’t work for three weeks, the impact is enormous. It would be easy to think I’m three weeks behind, that isn’t true, I didn’t do three weeks of development however in that period I would typically create ‘n’ new tickets and ‘m’ bugs.

I can never recover the lost time, that is gone, I’m going to use this as an opportunity to review the ‘plan’ and re-prioritise. As good as the ‘plan’ is I’m reasonably confident there are some ‘likes’ I can move further down the list and ‘needs’ that can move up the list.

How we expose Open Source REST API usage within our app

The Costs to Expect App is not an Open Source product; we intend to create a viable service; for now, we are keeping our secret sauce secret.

The App is built upon the Costs to Expect API, the API is Open Source and technically, anything we can do with the API, you can too. We aren’t gatekeeping your data; you can access your data through our App with the UI/UX we are creating, or, you can use other tools to fetch the data directly from the API.

To this end, we expose all GET, HEAD and OPTIONS requests, we hide POST, PUT, and PATCH requests as you can’t review them after the fact without unnecessary data being cached.

At the bottom of every page within the App, there is a table showing the required API requests. The table shows the request URI, the request METHOD, the response time, as well as was the request asynchronous or did we fetch it from our cache.

I appreciate that to the majority of users this data is redundant, yes there will be a visibility toggle; however, for the minority of users that are interested, I think the extra effort will be appreciated.

API requests table for the Costs to Expect App

Costs to Expect API v2.04.0 in development

House decorating continues, we expect to be finished sometime within the next two weeks. Decorating is limiting my development time; as such, I have decided it would be a bad idea to release v2.03.0.

If I release v2.03.0 now, it will probably be two weeks before v2.04.0 is ready. These releases are interim releases, combined they complete a significant feature. A two-week gap between the versions could cause issues so I’m releasing v2.03.0 internally and will make another public release when v2.04.0 is ready.

We aren’t due to start another round of decorating until February 2020, so typical development cadence should return in November.

v2.03.0 of the API is on the way.

The development of v2.03.0 is almost complete; I expect to have it out before the end of the week.

For this release, I am refactoring the multiple item types code. Adding support for simple expenses increased the item code significantly. New item types are due to be added to the API, so it makes sense to refactor before the complexity increases additionally.

We are decorating our house, and the scope of our plans increased, there is going to be a negative impact on my development time.

Although the pace of development has slowed, it is going to pay dividends in the future. I expect the pace to speed up once we complete decorating and I have pushed out v2.04.0 of the API.

The release is too big!

It isn’t unusual for me to release at least one new update a week for the Costs to Expect API and website. I find it significantly easier to manage and test with small iterative releases. As the only developer on this project, anything that makes my life simpler is a plus.

Occasionally a release has to touch a substantial part of the Costs to Expect API, v2.02.0 is an example of that. I need to review every section of code relating to items.

I intended to split the development work for multiple item types across two versions, v2.00.0 and v2.02.0. The database gets upgraded, and then I hit the code and expose the feature.

In planning, I appear to have overlooked summary routes. Rather than extend the design I developed to incorporate item summary routes; it makes sense to modify it slightly, given the ‘new’ requirements. If I soldiered on and upgraded the item summary routes, I suspect I would want to refactor them in a few weeks. Rather than face that situation, I feel it makes sense to go back to the drawing board and ensure I develop a design that will work going forward.

Post-release of v2.02.0, I will rework the current design, refactor as necessary and then add support for multiple item types to the item summary routes.

I gave myself one to two weeks to develop support for multiple item types. It turns out, two weeks is probably about right given the increased (actual) scope.

Three iterations of work left.

Upon checking my tracker this morning, I saw something that I haven’t seen since I began using Pivotal Tracker almost 1400 stories ago, the end of the backlog.

Am I done? No, far from, the last 14 months has all been phase one.

Unless I create additional tickets, which we both know I will*, phase two is due to begin at the end of October, the start of November.

I’ve made a point of not planning for phase two; I’m attempting to stop myself from getting distracted by future goals rather than the problems I need to solve now. I have been planning; I’ve just been trying to keep it all in my head and not put pen to paper.

In October I’ll start planning in earnest. Hopefully, the foundations I have created are suitable, I’m confident they are, but until I begin planning phase two, I will not know for sure.

*Alternatively, I could dive into the icebox and pull out a few tickets.

Costs to Expect roadmap

Now that version 2 of the API is out, it is time to publish a roadmap. It is not a comprehensive roadmap; it is merely a summary of where we are going with Costs to Expect over the next several months.

I work on numerous projects; the weeks match up approximately with when I expect to complete tasks, they don’t necessarily line up with the time required for the individual tasks.

Week 1

Minor fixes.

I’ll spend two to three days working through issues that arose during development.

Week 2-3

Multiple item types.

I’ll need to spend one to two weeks finishing all the changes necessary to support multiple item types. The database is ready; however, many parts of the API need to be updated to expose multiple item type support correctly.

Week 4

Permitted users.

I need to add permitted users management to the API. Once we have decided precisely how we want to limit user management, I need to implement it; this should not take more than a few days.

Week 5-7

App development

I’ll spend a couple of weeks building the base for the Costs to Expect App. The initial release will be private and have restricted user access; it will not be publically available.

I need to migrate the features from the web app, develop the base code for interacting with the API and work out the UX. I intend to build the majority of the forms for the app using the API Options requests, so I need to develop the relevant system.

Week 8-End of the year

Forecasting for the budgeting system.

I will spend the rest of the year planning and developing the forecasting system. The forecasting system touches both systems, the API and the App. We hope to have a version of the App ready for users at the end of the year.

We don’t know what will be in the first public release; it very much depends on how baked the features are; some may need a little more time. Costs to Expect is an evolving product; there will always be more coming.

Costs to Expect API v2.00.0, well v2.00.1

Yesterday was release day; I released v2.00.0 of the Costs to Expect API and pushed two minor updates to the Costs to Expect Website and Web app.

Other than two minor issues, the releases went better than I expected. I did release a small hotfix that fixes one of the problems as it was publically visible, the other will get resolved in v2.01.0.

I have two Postman monitors that run daily; I also run these regularly during development to try and ensure I don’t break anything. Collection one contains all the HTTP requests the Website makes; collection two is more thorough and has HTTP requests for nearly all API endpoints. The two collections include around 1200 tests, these test the HEADERS, returned status codes and the response body. There were no tests for the issues; they slipped through the net. I have added a ticket to my tracker to add tests to catch any possible regressions.

My Postman collections are far from perfect; they give me some confidence, but as is evident based on the v2.00.1 release, there are still gaps.

I’m going to work on improving the monitors, for the last six months, anytime a feature gets added to the API, a test gets added to the relevant collection. Testing is going to be especially important when I start developing the app in a few weeks.