Operational Processes
Support Hours and Responsibilities
Regular Support
On ordinary working days from 9AM to 4PM, at least one engineer will:
- Monitor and respond to queries in #ask-modernisation-platform
- Monitor #modernisation-platform-high-priority-alarms for incidents
- Check #modernisation-platform-notifications for build notifications and Dependabot PRs
- Triage new user support requests on the board
The whole support team will:
- Manage ongoing incidents
- Work on larger user requests and support backlog stories
Before ending the day (4PM), an engineer will hand over information about planned work or high-priority incidents to the on-call engineer.
Out-of-Hours Planned Changes Support
This process ensures that planned maintenance and updates are executed smoothly with appropriate support during out-of-hours periods.
Requesting Out-of-Hours Support
Advance Notice
Teams must provide a minimum of 48 hours notice for any planned changes requiring out-of-hours support.
Submission Process
Submit requests by creating a GitHub issue using the provided template. Use this link to access the template.
Once complete please notify the team via the #ask-modernisation-platform Slack channel once the issue is created.
Information to Include
- Change Details: A comprehensive description of the planned change.
- Scheduled Time: Proposed date and time for the change.
- Impact Assessment: Analysis of potential effects on services and users.
- Rollback Plan: A clear strategy for reverting changes if necessary.
Review Process
The Modernisation Platform team will assess the request, considering factors such as potential risks, resource availability, and alignment with ongoing activities.
Confirmation
Upon approval, confirmation will be communicated, including the assigned support personnel and any specific instructions.
Execution of Planned Change
Pre-Change Preparation
- Ensure all relevant stakeholders are informed.
- Verify that backup and rollback procedures are in place.
During Change Implementation
- The assigned support engineer will be available to assist as needed.
- Maintain open communication via a designated Slack channel for real-time updates.
Post-Change
- Confirm the success of the change in the designated Slack channel and monitor for any issues.
On-Call Support and Processes
Our on-call support is crucial for addressing critical issues outside of regular hours.
On-Call Hours
- Weekdays: 7AM–9AM and 4PM–10PM
- Weekends and Bank Holidays: 7AM-10PM
- On-Call support is not available outside of these hours
On-Call Rota
- Provided through Pagerduty
- Managed through code
- Consists of a primary engineer and a secondary engineer
- The primary engineer is the first point of contact
- The secondary engineer provides support to the primary as required
Responsibilities
On-call engineers will: - Respond to high-priority incidents during on-call hours - Work to resolve these incidents
If unable to resolve an issue, they will: - Inform the affected team - Develop a plan for resolution during normal working hours
Critical Incidents Procedure
For critical incidents outside of regular hours: - Email contact-on-call-modernisation-platform@digital.justice.gov.uk - Include: - Known facts about the incident - Slack channel for further communication - The on-call person will respond via the specified Slack channel as soon as possible
Note: This process is ONLY for critical incidents and not for routine out-of-hours application or development support.
Communication Channels
#ask-modernisation-platform
The #ask-modernisation-platform channel is our main entry point for support. The main purpose of the channel is to discuss the problems that people are having and help them to solve them.
We encourage people to ask questions and report problems in this channel and we’ll do the best we can to help. We also welcome comments, opinions and any feedback regarding any of the services and the platform as a whole. This channel can also be used to communicate to other Modernisation Platform users on topics you think that might be useful for other users.
One of the Modernisation Platform team will be available to answer questions throughout the hours of support (9AM - 4PM).
#modernisation-platform-update
#modernisation-platform-update is a low-volume channel for broadcast communications.
Examples include:
- Announcements of future scheduled work
- Information about planned maintenance
- Updates on ongoing incidents
In these situations:
When posting announcements: - Use @here and @channel sparingly - Consider using visual cues to highlight announcements - Commit to and maintain regular update schedules
Posts in this channel should also be forwarded to #ask-modernisation-platform.
Our on call process
Team members who are on call manage an on call rota in PagerDuty. The on call rota consists of a primary engineer and a secondary engineer. The primary engineer is the first point of contact. The Modernisation Platform is complex so we have a secondary engineer on call. Their role is to provide support to the primary if this is required. Team members that are new to the rota should ensure that they have a secondary who has enough knowledge and experience of the process to support them.
The on-call hours are 7AM–9AM and 4PM–10PM weekdays, 7AM-10PM weekends and bank holidays.
On call team members will respond to high priority incidents during on call hours and will work on those incidents. If the engineer is not able to resolve the issue they will:
- Inform the affected team
- Put together a plan to resolve the issue during normal working hours
Critical incidents only - contact the out-of-hours on-call person
In the event of a ‘Critical incident’ to the Modernisation Platform infrastructure affecting applications (out of support hours), please email contact-on-call-modernisation-platform@digital.justice.gov.uk, stating:
- The known facts surrounding the incident
- Please also state the ‘Slack’ channel on which further communication can take place.
The on-call person will get back to you on ‘Slack’ as soon as possible. It must be emphasised that this is ONLY for ‘Critical Incidents’ and is not for out-of-hours application or development support.
Documentation
We maintain documentation in all of our GitHub repositories.
The Modernisation Platform repository is the starting point for our documentation.
The Modernisation Platform User Guideis our front door for users, containing concepts, guides, runbooks, and other helpful information.