PATCH support…finally.

I added PATCH support to the Costs to Expect API in November. At the time I did the minimum, I only added PATCH support for items as that was the only data we needed to edit.

In March, I reviewed the PATCH code, making a few minor alterations.

It is now time to add PATCH support to the rest of the API. PATCH support will be added for resource types, resources, categories, subcategories and category assignments for items.

When I reviewed the PATCH code in March, I was looking at a single instance. This time I need to implement PATCH support across multiple routes; we will see how many changes I make in attempting to make the code DRY.

I have six tickets in my tracker, one for each route plus one additional chore for minor refactoring. If everything goes well, I should have a release ready in a few days, and I shouldn’t have created any extra refactoring chores.

Slight delay to phase two

I had planned to start developing the Costs to Expect service during the summer, as always, there be a slight delay. We decided that rather than steam ahead with new development, it made sense to deal with some of the technical debt.

Looking at the planner, I’ve got somewhere between two and four weeks of development to complete depending on how much debt we decide to pay back.

As it stands today, our release plans are as below;

Costs to Expect API

  • v1.21.0 – Similar to the latest release, prep work and refactoring.
  • v1.22.0 – PATCH support for categories, subcategories, resource types and resources.
  • v1.23.0 – Summaries for categories, subcategories, resource types and resources
  • v1.24.0 – Add support for permitted users.

Costs to Expect website

  • v1.10.0 – General housekeeping and prep.
  • v1.11.0 – Authentication

Phase two of the Costs to Expect service should begin after the above is complete. Phase two is a migration project; the functionality that exists in the web-app needs to be migrated to the website. My Wife will then be able to use the website full time, and the web-app can be taken offline.

Override pagination limits

Pagination is necessary; you don’t want your API/website/app to default to returning everything from the database, occasionally though you might want to fetch the entire collection.

On the Costs to Expect API, we have added a collection parameter, if set to true the pagination limits do not apply, and the entire collection is requestable.

There are a couple of caveats, the API defaults the parameter to false, and we only allow the parameter for collections which will know will be limited in size.

PHPStorm, Docker and Xdebug

I use Docker for development; on the whole, I love it; however, getting a couple of things set up proved a little challenging, namely debugging from PHPStorm with Xdebug.

Eventually, I will update my Docker quick start project on GitHub, however for now, here are some basic instructions.

We need a couple of arguments in our .env file;

 WITH_XDEBUG=true
PHP_IDE_CONFIG=serverName=[name-of-server-in-IDE]
XDEBUG_CONFIG=remote_host=host.docker.internal

Next, we update the Dockerfile

ARG WITH_XDEBUG=false

RUN if [ $WITH_XDEBUG = "true" ] ; then \
pecl install xdebug; \
docker-php-ext-enable xdebug; \
echo "error_reporting = E_ALL" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "display_startup_errors = On" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "display_errors = On" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
echo "xdebug.remote_enable=1" >> /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini; \
fi ;

Finally, on the Docker side, we need to update our docker-compose file.

 services:
yourservice:
build:
args:
- WITH_XDEBUG=${WITH_XDEBUG}
env_file: .env

Set up your server, making sure to choose your debugger, Xdebug in this case. Settings -> Languages & Frameworks -> PHP -> Servers. The name of the server should match the name you set in the .env file.

Set the debug port and enable “can accept external connections”, PHPStorm, settings -> Languages & Frameworks -> PHP -> Debug, the port defaults to 9000.

Add a run/debug configuration (Run -> Edit configurations), select your server and optionally set an IDE key. To ensure everything is set correctly, you can use the validate option to check your configuration.

Start listening, add a breakpoint and away you go, you can now debug an app running in Docker.

Summer of fun

If I maintain the same development pace as the last eight weeks, I’m due to have a “Summer of fun”.

I’m two weeks away, one development iteration from a point in which I’ll be comfortable letting the Costs to Expect API and website run throughout the summer untouched.

They won’t be untouched; both the website and API are under active development; I’ll focus my energy on building the service, not the public website.

Dependant on your experience, as a developer, you don’t typically get to spend too much time working on new projects solving problems entirely new to you. Mid-way through July, that is me, I’m planning on locking myself in my office and getting down and dirty with my IDE, whiteboard, pen and paper.

My wife and children have told me that they have a very different idea on what a “Summer of fun” is; I’m guessing it is far less coding than my plan.

Almost one year in, the fun begins

The Costs to Expect API is approaching the point where it includes all the standard and expected features; you can add items, edit, list, delete and summarise the data, all the features you would expect at a minimum from a basic Rest API.

This summer, the fun begins, almost one year after development began I get to start work on the non-standard features and start developing the Costs to Expect service.

It took a long time to get here, well, a year. One year ago, I had an idea; now there is a Rest API, three web apps that use the API and a shiny new website showcasing five years of data that we have collected.

I know what we have planned for the next couple of months, and I know the long term goals, it is going to be interesting to look back again in 12 months to see how much progress there was and how much we still have to do.

I’ll blog more about the specific features when I start development, during the summer, I expect progress will slow slightly as I spend time with my family.

Costs to Expect API – https://api.costs-to-expect.com
Costs to Expect Website – https://www.costs-to-expect.com
Costs to Expect on GitHub – https://github.com/costs-to-expect

API endpoints, filtering, sorting, searching, limiting and summarising

The Costs to Expect API is developing slowly; features get added as I need them, v1.15.0 is an exciting release, it will be the first time an endpoint in the API can be filtered, sorted, limited, searched and summarised.

I’ve written previously about summarising collections, with the release of search in v1.15.0 it makes send to describe all options supported by the items (/v1/resource-types/[resource-type]/resources/[resource]/items) collection on the Costs to Expect API.

By default, the items collection returns all child expenses in reverse chronological order, ‘all’ and ‘reverse chronological’ are sensible defaults; however, the API is limited if the collection always returns everything, you need to be able to filter, sort, limit, search and summarise.

The Costs to Expect API supports filtering, sorting, limiting and searching via parameters appended to the URI, collections are summarised by prepending /summary/ to the URI. Summary endpoints support the same filtering and searching options as their companion collections, limiting and sorting don’t make any sense in the context of a summary.

A URI can quickly become unreadable when multiple GET parameters are assigned, to attempt and improve readability I have grouped sorting and searching.

The OPTIONS requests for the Costs to Expect API detail all the supported GET parameters, in addition to giving an overview of each parameter, the OPTIONS requests list which fields and sortable and searchable. The sort and search parameters both allow multiple conditions.

Limiting a collection is handled as expected, there is an offset parameter to define the start record and a limit parameter to define the number of records requested.

Filtering a collection is handled via individual parameters, filtering is more common than searching and in my opinion needs to be the most verbose, when reviewing a URI the filtering parameters should be more visible.

This solution won’t work for everyone; however, if you need to support filtering, searching, sorting, limiting and summarising, I think it provides an excellent level or readability while still being practical.

Slowly developing Costs to Expect

I’m developing Costs to Expect in a slightly different manner to any of my previous large personal projects. The process is slightly different for two reasons, firstly, the design, secondly, I’m deliberately developing at a slower, more iterative pace.

Design.

There are two parts to Costs to Expect, the Open Source API and the website, the website will include the secret sauce for the service.

The API has not been explicitly written for the Costs to Expect website; the intention has always been to use it for separate projects and idealy give back to the community, hence being Open Source. So far, I have one other project that uses the API, and I have plans for another rattling around my head. I’m confident that eventually, tomorrow maybe, I will add a feature to the API specifically for the website. I intend to try and ensure features are only added to the API if they make sense in the context of the API.

Slow, iterative development.

I released the API during the summer and developed a simple app to allow my wife to enter expenses for our children; the API is not anywhere near complete, I am continually fleshing out the features priotising the features I need to develop the website, the latest example is sorting. The web app my wife uses does not require sorting, the public Costs to Expect website does; that is why I’m adding it now.

There are three initial phases of the development of the website; once all three are complete, the real project begins.

Phase 1: The website consumes the API and shows the current expenses for our children in an engaging way for public consumption.
Phase 2: My Wife is able to transition from using the web app I developed in the summer; she should be able to do the expenses management for our children using the website.
Phase 3: I develop the forecasting and budgeting systems and start opening up the website to a restricted number of test users.

I’m very much in Phase 1. I envision I have at least another four weeks of development before I can progress to Phase 2.

Phase 2 isn’t complicated; I need to ensure I make the correct decisions, I need to work out my authentication, probably oAuth, how I’m going to handle users, groups and then develop the management pages.

Phase 3 will be when the fun begins and the part I’m most looking forward to reaching. Phases 2 and 3 require time; I mostly know what is needed; I need to design and then code the systems.

The majority of my year off will be in Phase 3, I know what I need to end up with, however, I’m currently not quite sure how I’m going to get there.

For each Phase, I’m developing very slowly; the visual design is mostly solved thanks to a highly skilled designer I employed late last year, I mostly only need to think about what I need and how to develop it.

As a single developer with limited development time and resource, I can’t afford to waste any of my development time. I’m prototyping every feature and then slowly developing them, as an example, the website won’t be connecting to the API until at least v1.04.0, up until then I’ll be working on the responsive layout, building the view components I require and experimenting with the UX.

Year off

Eight weeks ago I concluded my 16th contract, yes, I have been contracting for a while, over ten years so far.

I enjoy contracting; I continually meet new people; additionally, by moving around, I have been fortunate enough to be exposed to numerous business domains, frameworks and tools.

We can’t ignore the elephant in the room, contracting pays well, especially when you have over 20 years of experience.

In April, my Wife had our second child, Niall. When we had our first child, we realised that the best way to provide the support my Wife needed was to take time off, there were so many positives with me providing the support rather than getting in professional help. Well, with Niall, we are going to do the same thing, I’m going to take at least a year off, primarily to help my Wife raise our brand-new son, secondly, to try and get the Costs to Expect service off the ground.

Costs to Expect is a project I had been considering for a while, during the summer I released the first version of the API; I plan to develop the MVP of the service during my time off.

Twelve months is not as long as you think, especially when you are a single developer. I’ve used Pivotal Tracker for years; it has become scarily accurate at predicting what I will be able to accomplish during a two-week sprint. I usually have two or three sprints worth of work in my backlog, according to Pivotal, it is going to be quite difficult to get enough of the service ready in 12 months, hopefully, if I plan well, I should be able to do it.

API Summary routes

In a previous blog post, I outlined my solution to summary routes, in short, every route in my API will have a matching summary route, the summary route is simply the route with a summary/ prefix.

If the GET endpoint of a route supports parameters the summary route should support the same parameters, in my experience, it will need additional parameters; as always, best shown with an example.

The summary route lives at summary/resource-types/[resource-type]/resources/[resource]/items, the GET endpoint has eight parameters, below I’ll show the expected output for each parameter.

In my API I have the following route /resource-types/[resource-type]/resources/[resource]/items. The GET endpoint returns a collection of the items assigned to the resource; it has four parameters for filtering, year, month, category and subcategory.

No parameters | Total sum of items
years | Summary for each year
year | Summary for the requested year
months | Summary for each month
month | Summary for the requested month
categories | Summary for each category
category | Summary for the requested category
subcategories | Summary for each subcategory
subcategory | Summary for the requested subcategory

So far I’ve not come across any negatives to this structure, yes, there are numerous parameters for the GET endpoint, I don’t consider that an issue, I’d rather have fewer routes than fewer GET parameters.