The release is too big!

It isn’t unusual for me to release at least one new update a week for the Costs to Expect API and website. I find it significantly easier to manage and test with small iterative releases. As the only developer on this project, anything that makes my life simpler is a plus.

Occasionally a release has to touch a substantial part of the Costs to Expect API, v2.02.0 is an example of that. I need to review every section of code relating to items.

I intended to split the development work for multiple item types across two versions, v2.00.0 and v2.02.0. The database gets upgraded, and then I hit the code and expose the feature.

In planning, I appear to have overlooked summary routes. Rather than extend the design I developed to incorporate item summary routes; it makes sense to modify it slightly, given the ‘new’ requirements. If I soldiered on and upgraded the item summary routes, I suspect I would want to refactor them in a few weeks. Rather than face that situation, I feel it makes sense to go back to the drawing board and ensure I develop a design that will work going forward.

Post-release of v2.02.0, I will rework the current design, refactor as necessary and then add support for multiple item types to the item summary routes.

I gave myself one to two weeks to develop support for multiple item types. It turns out, two weeks is probably about right given the increased (actual) scope.

Three iterations of work left.

Upon checking my tracker this morning, I saw something that I haven’t seen since I began using Pivotal Tracker almost 1400 stories ago, the end of the backlog.

Am I done? No, far from, the last 14 months has all been phase one.

Unless I create additional tickets, which we both know I will*, phase two is due to begin at the end of October, the start of November.

I’ve made a point of not planning for phase two; I’m attempting to stop myself from getting distracted by future goals rather than the problems I need to solve now. I have been planning; I’ve just been trying to keep it all in my head and not put pen to paper.

In October I’ll start planning in earnest. Hopefully, the foundations I have created are suitable, I’m confident they are, but until I begin planning phase two, I will not know for sure.

*Alternatively, I could dive into the icebox and pull out a few tickets.

Costs to Expect roadmap

Now that version 2 of the API is out, it is time to publish a roadmap. It is not a comprehensive roadmap; it is merely a summary of where we are going with Costs to Expect over the next several months.

I work on numerous projects; the weeks match up approximately with when I expect to complete tasks, they don’t necessarily line up with the time required for the individual tasks.

Week 1

Minor fixes.

I’ll spend two to three days working through issues that arose during development.

Week 2-3

Multiple item types.

I’ll need to spend one to two weeks finishing all the changes necessary to support multiple item types. The database is ready; however, many parts of the API need to be updated to expose multiple item type support correctly.

Week 4

Permitted users.

I need to add permitted users management to the API. Once we have decided precisely how we want to limit user management, I need to implement it; this should not take more than a few days.

Week 5-7

App development

I’ll spend a couple of weeks building the base for the Costs to Expect App. The initial release will be private and have restricted user access; it will not be publically available.

I need to migrate the features from the web app, develop the base code for interacting with the API and work out the UX. I intend to build the majority of the forms for the app using the API Options requests, so I need to develop the relevant system.

Week 8-End of the year

Forecasting for the budgeting system.

I will spend the rest of the year planning and developing the forecasting system. The forecasting system touches both systems, the API and the App. We hope to have a version of the App ready for users at the end of the year.

We don’t know what will be in the first public release; it very much depends on how baked the features are; some may need a little more time. Costs to Expect is an evolving product; there will always be more coming.

Costs to Expect API v2.00.0, well v2.00.1

Yesterday was release day; I released v2.00.0 of the Costs to Expect API and pushed two minor updates to the Costs to Expect Website and Web app.

Other than two minor issues, the releases went better than I expected. I did release a small hotfix that fixes one of the problems as it was publically visible, the other will get resolved in v2.01.0.

I have two Postman monitors that run daily; I also run these regularly during development to try and ensure I don’t break anything. Collection one contains all the HTTP requests the Website makes; collection two is more thorough and has HTTP requests for nearly all API endpoints. The two collections include around 1200 tests, these test the HEADERS, returned status codes and the response body. There were no tests for the issues; they slipped through the net. I have added a ticket to my tracker to add tests to catch any possible regressions.

My Postman collections are far from perfect; they give me some confidence, but as is evident based on the v2.00.1 release, there are still gaps.

I’m going to work on improving the monitors, for the last six months, anytime a feature gets added to the API, a test gets added to the relevant collection. Testing is going to be especially important when I start developing the app in a few weeks.

Version 2.00.0 on the way

The initial development work for v2.00.0 of the Costs to Expect API is almost complete. I’m just one dev session away from all 1200 Postman tests passing.

I used the word initial, the v2.00.0 branch works; however, the two core features are not going to be complete until a couple/few .01 releases after v2.00.0. The base v2 version is about getting the database ready and ensuring all the migrations are valid; it is a significant change to the v1 database.

Permitted users are pretty much complete, the API will be missing the management of users, but that will quickly appear in .01 release.

Multiple item types, this is the big one.

As mentioned above, the database work will be complete; I should be able to write simple migration files for all additional releases without affecting existing data. The code is a little bit of a clutz; I’m merely joining to the only known item type sub table.

Updating all queries with a join to the item_type_allocated_expense table works; however, the code isn’t dynamic. When additional item types get added, creating, deleting, sorting and searching will all fail.

Once I am happy with the initial v2.00.0 release, I will add permitted users management and then work out how I am going to refactor the code to support dynamic item types. I have several ideas; however, I need to come up with something scalable and going forward will work for what I have planned for v2, v3 and beyond.

I had planned the first (private) release of the Costs to Expect app to be in late September, given the additional work, I suspect it will now get bumped into October.

We should still have something ready for public consumption before the end of the year. We are experiencing a minor bump in the road; these issues aren’t anything I wasn’t going to have to work out anyway, I have to solve some interesting problems sooner rather than later.

Costs to Expect API, v1.24 -> v2.00

Before I start developing the Costs to Expect app, I need to enhance the Costs to Expect API. Two significant features are going to be in the next version of the API, permitted users and multiple items types.

Permitted users

The Costs to Expect API supports authenticated users; however, the assumption is all authenticated users have the same data access. This system works at the moment because there are only two users; this works at the moment as there are only two users. The app needs to support multiple users with their own private data, so I’m going through the app and adding the necessary user links.

Multiple item types

The `item` table is set up for child expenses; that was all I needed for the initial development of the API. Looking forward, the API needs to support more complex item types. In the next release, I’m going to add support for multiple items types with a long term goal being support for custom item types.

The item type will need to be selected when you create a resource type. If you are tracking expenses you will choose `expense item type`, for budgeting you would choose `budget expense type`, and for a list, you would select `list item type`.

Custom item types will be coming; at this time, I can’t confirm when. I need to develop the budgeting and forecasting system; after that, I will be able to give a rough timeline.

Version 2.00

The features above are significant, far more significant than any additions to the API since its release in 2018. I feel that these additions are enough to justify a v2.00 release. 

A bump in the major version means I’m not going to support upgrades from v1.24 to v2.00. Upgrading from v1 to v2, will be possible. To upgrade you will need to export and import your existing data into a new instance of the API. I suspect there will be a join/relationship table between `items` and their data. Until v2.00 of the API is out, I am unable to confirm the exact setup; it is still in development.

Custom item types are unlikely to be supported in the v2.00 branch, I suspect, they will be the notable feature for v3.00. Unlike the jump from v1 to v2, I can guarantee there will be an upgrade path from v2 to v3, the existence of the app and website (Costs to Expect service) will require it.

Open Source Costs to Expect website

I have decided to Open Source the Costs to Expect website.

I planned the website and the app for the service as a single app. Over time, I’ve concluded that although it may create additional work, it makes sense to separate the service app (app.costs-to-expect.com) from the public website (www.costs-to-expect.com).

Due to the limited requirements, the public website will never be an excellent showcase of the Costs to Expect service.

Until we decide I need help, I’m creating the Costs to Expect service by myself. Managing three apps is more involved than two; however, there are pluses to separating the products.

The public website is a minor part of the service; I have numerous features planned; however, they are much lower in priority than developing the app. I don’t want to be in a position whereby the website is restricting the development of the app.

For the next three to four months, I am moderately confident that I can develop the app and update the API without negatively affecting the website. I have request tests in place (thank you Postman) that will immediately notify me if I make any change to the API, which affects the website.

PATCH support…finally.

I added PATCH support to the Costs to Expect API in November. At the time I did the minimum, I only added PATCH support for items as that was the only data we needed to edit.

In March, I reviewed the PATCH code, making a few minor alterations.

It is now time to add PATCH support to the rest of the API. PATCH support will be added for resource types, resources, categories, subcategories and category assignments for items.

When I reviewed the PATCH code in March, I was looking at a single instance. This time I need to implement PATCH support across multiple routes; we will see how many changes I make in attempting to make the code DRY.

I have six tickets in my tracker, one for each route plus one additional chore for minor refactoring. If everything goes well, I should have a release ready in a few days, and I shouldn’t have created any extra refactoring chores.

Slight delay to phase two

I had planned to start developing the Costs to Expect service during the summer, as always, there be a slight delay. We decided that rather than steam ahead with new development, it made sense to deal with some of the technical debt.

Looking at the planner, I’ve got somewhere between two and four weeks of development to complete depending on how much debt we decide to pay back.

As it stands today, our release plans are as below;

Costs to Expect API

  • v1.21.0 – Similar to the latest release, prep work and refactoring.
  • v1.22.0 – PATCH support for categories, subcategories, resource types and resources.
  • v1.23.0 – Summaries for categories, subcategories, resource types and resources
  • v1.24.0 – Add support for permitted users.

Costs to Expect website

  • v1.10.0 – General housekeeping and prep.
  • v1.11.0 – Authentication

Phase two of the Costs to Expect service should begin after the above is complete. Phase two is a migration project; the functionality that exists in the web-app needs to be migrated to the website. My Wife will then be able to use the website full time, and the web-app can be taken offline.

Override pagination limits

Pagination is necessary; you don’t want your API/website/app to default to returning everything from the database, occasionally though you might want to fetch the entire collection.

On the Costs to Expect API, we have added a collection parameter, if set to true the pagination limits do not apply, and the entire collection is requestable.

There are a couple of caveats, the API defaults the parameter to false, and we only allow the parameter for collections which will know will be limited in size.