Summer of fun

If I maintain the same development pace as the last eight weeks, I’m due to have a “Summer of fun”.

I’m two weeks away, one development iteration from a point in which I’ll be comfortable letting the Costs to Expect API and website run throughout the summer untouched.

They won’t be untouched; both the website and API are under active development; I’ll focus my energy on building the service, not the public website.

Dependant on your experience, as a developer, you don’t typically get to spend too much time working on new projects solving problems entirely new to you. Mid-way through July, that is me, I’m planning on locking myself in my office and getting down and dirty with my IDE, whiteboard, pen and paper.

My wife and children have told me that they have a very different idea on what a “Summer of fun” is; I’m guessing it is far less coding than my plan.

API endpoints, filtering, sorting, searching, limiting and summarising

The Costs to Expect API is developing slowly; features get added as I need them, v1.15.0 is an exciting release, it will be the first time an endpoint in the API can be filtered, sorted, limited, searched and summarised.

I’ve written previously about summarising collections, with the release of search in v1.15.0 it makes send to describe all options supported by the items (/v1/resource-types/[resource-type]/resources/[resource]/items) collection on the Costs to Expect API.

By default, the items collection returns all child expenses in reverse chronological order, ‘all’ and ‘reverse chronological’ are sensible defaults; however, the API is limited if the collection always returns everything, you need to be able to filter, sort, limit, search and summarise.

The Costs to Expect API supports filtering, sorting, limiting and searching via parameters appended to the URI, collections are summarised by prepending /summary/ to the URI. Summary endpoints support the same filtering and searching options as their companion collections, limiting and sorting don’t make any sense in the context of a summary.

A URI can quickly become unreadable when multiple GET parameters are assigned, to attempt and improve readability I have grouped sorting and searching.

The OPTIONS requests for the Costs to Expect API detail all the supported GET parameters, in addition to giving an overview of each parameter, the OPTIONS requests list which fields and sortable and searchable. The sort and search parameters both allow multiple conditions.

Limiting a collection is handled as expected, there is an offset parameter to define the start record and a limit parameter to define the number of records requested.

Filtering a collection is handled via individual parameters, filtering is more common than searching and in my opinion needs to be the most verbose, when reviewing a URI the filtering parameters should be more visible.

This solution won’t work for everyone; however, if you need to support filtering, searching, sorting, limiting and summarising, I think it provides an excellent level or readability while still being practical.

Slowly developing Costs to Expect

I’m developing Costs to Expect in a slightly different manner to any of my previous large personal projects. The process is slightly different for two reasons, firstly, the design, secondly, I’m deliberately developing at a slower, more iterative pace.

Design.

There are two parts to Costs to Expect, the Open Source API and the website, the website will include the secret sauce for the service.

The API has not been explicitly written for the Costs to Expect website; the intention has always been to use it for separate projects and idealy give back to the community, hence being Open Source. So far, I have one other project that uses the API, and I have plans for another rattling around my head. I’m confident that eventually, tomorrow maybe, I will add a feature to the API specifically for the website. I intend to try and ensure features are only added to the API if they make sense in the context of the API.

Slow, iterative development.

I released the API during the summer and developed a simple app to allow my wife to enter expenses for our children; the API is not anywhere near complete, I am continually fleshing out the features priotising the features I need to develop the website, the latest example is sorting. The web app my wife uses does not require sorting, the public Costs to Expect website does; that is why I’m adding it now.

There are three initial phases of the development of the website; once all three are complete, the real project begins.

Phase 1: The website consumes the API and shows the current expenses for our children in an engaging way for public consumption.
Phase 2: My Wife is able to transition from using the web app I developed in the summer; she should be able to do the expenses management for our children using the website.
Phase 3: I develop the forecasting and budgeting systems and start opening up the website to a restricted number of test users.

I’m very much in Phase 1. I envision I have at least another four weeks of development before I can progress to Phase 2.

Phase 2 isn’t complicated; I need to ensure I make the correct decisions, I need to work out my authentication, probably oAuth, how I’m going to handle users, groups and then develop the management pages.

Phase 3 will be when the fun begins and the part I’m most looking forward to reaching. Phases 2 and 3 require time; I mostly know what is needed; I need to design and then code the systems.

The majority of my year off will be in Phase 3, I know what I need to end up with, however, I’m currently not quite sure how I’m going to get there.

For each Phase, I’m developing very slowly; the visual design is mostly solved thanks to a highly skilled designer I employed late last year, I mostly only need to think about what I need and how to develop it.

As a single developer with limited development time and resource, I can’t afford to waste any of my development time. I’m prototyping every feature and then slowly developing them, as an example, the website won’t be connecting to the API until at least v1.04.0, up until then I’ll be working on the responsive layout, building the view components I require and experimenting with the UX.

API Summary routes

In a previous blog post, I outlined my solution to summary routes, in short, every route in my API will have a matching summary route, the summary route is simply the route with a summary/ prefix.

If the GET endpoint of a route supports parameters the summary route should support the same parameters, in my experience, it will need additional parameters; as always, best shown with an example.

The summary route lives at summary/resource-types/[resource-type]/resources/[resource]/items, the GET endpoint has eight parameters, below I’ll show the expected output for each parameter.

In my API I have the following route /resource-types/[resource-type]/resources/[resource]/items. The GET endpoint returns a collection of the items assigned to the resource; it has four parameters for filtering, year, month, category and subcategory.

No parameters | Total sum of items
years | Summary for each year
year | Summary for the requested year
months | Summary for each month
month | Summary for the requested month
categories | Summary for each category
category | Summary for the requested category
subcategories | Summary for each subcategory
subcategory | Summary for the requested subcategory

So far I’ve not come across any negatives to this structure, yes, there are numerous parameters for the GET endpoint, I don’t consider that an issue, I’d rather have fewer routes than fewer GET parameters.

Dart Sass with Windows Linux Subsystem

Follow the steps below to setup sass, specifically dart-sass in the Windows Linux Subsystem.

  1. Download and extract the dart-sass release you are going to use, at the time of this post I opted for dart-sass-1.18.0-linux-x64.tar.gz
  2. Find your bash.rc, if you setup WLSS after the Fall Creators update it will be located at C:\Users\USERNAME\AppData\Local\Packages\{CanonicalGroupLimited.UbuntuonWindows_...}\LocalState\rootfs\home\LINUXUSER\ or similar.
  3. Add export PATH="/path/to/dart-sass:$PATH" to the end of the file, in my case export PATH="/mnt/c/Users/USERNAME/Documents/GitHub/dart-sass:$PATH"

Open bash, type sass --version to see if everything worked.

Where to put summary routes in your API?

I believe the response to the question is going to be different for every API, in my case I initially added them where I thought it made sense, at the level I thought I wanted to summarise.

To view the list of items assigned to a resource in the Costs to Expect API you browse as below;
/resource-type/[id]/resource/[id]/items.
To view the TCO (Total cost of ownership) for a resource, I added a summary route at /resource-type/[id]/resource/[id]/summary/tco.

This initially made sense, however, later, as I was adding year and month summaries my solution began to become unwieldy, longterm, I would end up with two mismatching trees, the main tree for the API and then another summary tree, secondly, using this structure, where would I put a summary for multiple resources?

I’ve come up with a solution that I think will solve my problems, there should be a summary route for every API endpoint, the summary routes are then merely the route prefixed with /summary.

In the case of the TCO for a resource, the summary route would be summary/resource-type/[id]/resource/[id]/items. No TCO in the URI, it is not necessary, you are summarising the items collection so you should expect a total.

My solution doesn’t fix all the issues, presently the annual summary for a resource lives at /resource-type/[id]/resource/[id]/summary/years. There is no matching endpoint for this summary route. The solution, GET parameters. The items collection has four filtering parameters, year, month, category and subcategory; the summary route should support the same parameters.

I’m confident that if I had spent a little more time researching I would have been able to find this solution in an article somewhere online, I didn’t and unfortunately, it took me a little while to realise. Hopefully, this blog post will help at least one other person.

Do not change URIs, oops

A core rule of the Internet, don’t change a URI. I’m going to change some, why, we all make mistakes.

I released the initial version of the Costs to Expect API during the summer of ’18. It turns out when you review your code/API after a short break you spot all the issues you were oblivious to during development.

Over the next 12 months, I’m going to extend the Costs to Expect API, an API that I intend on maintaining for a significant period; I need to record data for at least the next 13 years.

If there were one small issue with the URIs, I’d deal with it, change a couple of URIs and add redirects. There are numerous issues; I am not happy with the initial summary routes. I favour dashes in the URIs over underscores, some of the words are incorrect and other minor issues.

We haven’t pushed the service; it doesn’t exist. As far as I am aware I am the only consumer of the API. I believe it is OK to modify the URIs, as long as I do not modify them again they should remain the same for twenty times longer then they have existed.

Self-documenting APIs

During my professional career I’ve worked on many APIs, however, I’ve never been responsible for the design, development and support of an API of more than trivial scale, when designing my own I wanted to do ‘as good of a job’ as I possibly can.

I’ve heard the term self-documenting APIs and never really looked into it, regardless, I’m going to describe what it means to me, hey, it is the internet, we all have opinions.

To me, there are five points.

  • The initial entry route or an obviously named route should display all the endpoints.
  • The initial entry route or an obviously named route should show the current version of the API as well as provide links to the README, CHANGELOG and anything else useful.
  • There should be a changelog route to describe all changes and updates to the API.
  • An OPTIONS request should exist for every route. The OPTIONS request should detail the purpose of the route, all possible verbs, as well as any fields and parameters.
  • No redundant information in either the payload or the response, for example, the payload should not be wrapped in an envelope, the verb is the envelope.

As a bonus, API versioning should be controlled via the route, any payloads or responses should be free of any version information.

OPTIONS requests

When I published the Costs to Expect API I was initially bemused, none of my OPTIONS requests worked, they all return “405 method not allowed”.

After searching, I diagnosed the problem. The HTTP OPTIONS request fails because the default PHP-CGI handler does not handle the “OPTIONS” verb.

The fix is simple, update or add the handler in web.config.

<handlers>
<remove name=”PHP72_via_FastCGI” />
<add name=”PHP72_via_FastCGI” path=”*.php” verb=”GET,PUT,POST,DELETE,HEAD,OPTIONS” modules=”FastCgiModule” scriptProcessor=”PATH\To\File\php-cgi.exe” resourceType=”Either” requireAccess=”Script” />
</handlers>

Releasing…works

I have a long-term project, Dlayer, that I very much believe in, but after many false starts, it still doesn’t exist, it will though, I’m due to restart it later this year with a slightly different focus 🙂

Why did I never finish or release the project? I never managed to hit what I considered to be the minimum viable product (MVP), my idea of the MVP for Dlayer was too vast, and far too much for one developer, I never managed to get enough done.

Early last year I needed to develop a small library, my PHP Quill renderer, I developed just what I needed at the time and quickly released it, within six months I bumped it to v1.00. I knocked it to v1.00 because I was confident in the library, it had warts, but people seemed to be using it, it wasn’t just solving my personal problem.

I’m not suggesting tagging it with a v1.00 release number made it slightly popular, I’m saying that because I was confident enough to add the v1.00 release number it became more popular.

Tagging the library with v1.00 meant I had to handle it differently, I needed to ensure the README and CHANGELOG were accurate, and up to date, tests are required, and I needed to learn how to handle the pull requests I started receiving. All of this made the library better, as I’ve said in the past, nothing makes your code better than knowing there is a minuscule chance other developers will look at your library.

During the summer I released two apps, both v1.00, the REST API for Costs to Expect and the companion Web app. Neither of these apps is complete. The REST API is missing quite a few features; however, it supports enough to get the job done, replace my spreadsheet and allow my wife to submit data.

I’m not saying anything new; there is a blog post by Jeff Atwood referencing a blog post by Chuck Jazdzewski stating our job is to ship. Knowing that is one thing, doing it is another, I wish I had had the confidence to ship earlier, and I hope I keep it up.