Despite the current tumutlous events, the redevelopment of the new OASIS form has continued. In March 2020, the ‘Beta’ system was demonstrated (remotely) to the OASIS Management Board for comment. Although a few areas still need finessing (as with any IT development), the core infrastructure has been completed. We are now beginning the transition from a Beta (effectively a working draft in this case) to a fully live system that replaces the current form.
In the various information you may have seen, the new version of OASIS is sometimes synonymous with HERALD (a project name). However, we have always been sticking with the nameOASIS for the system itself. The current version everyone is using now (as I write in April 2020) is the fourth major iteration deployed since 2002. Internally, we often refer to it as OASIS IV. It is important to note that throughout all subsequent discussion we’re simply going to refer to the new version as OASIS V.
In many ways. moving between versions is more complex than ‘just’ code development: OASIS has over 88,000 records , and over 5000 unique users representing 1386 organisations. It’s also used intensively.
In 2019 there were 108,000+ requests for pages within the OASIS form. We don’t know what these are though (could be one person clicking through alot of times!).
In 2019 there were 19919 edits to forms that resulted in a new version
We simply can’t suddenly move everyone over in a week! We need to gauge performance, iron out some issues, and start familiarising as many people as we can with how the new system differs from the old one. Most importantly, we don’t want a user to be inputting data into two systems and duplicating effort. This is the next phase: “Rollout of OASIS V”.
Timeline for Rollout of OASIS V
The following is the planned timeline for the next phase. It is important to note, that unless directed otherwise over this period, use of OASIS IV (the current form) should continue until the Autumn of 2020.
HER familiarisation with OASIS V Internal testing Review of OASIS V stability under increased use
Development Sprint Testing and Feedback with OASIS V Working Group
Silver release of OASIS V: a major redeployment Moving a small number of early adopters over to OASIS V
Development Sprint Testing and Feedback with OASIS Working Group
Development Sprint Testing and Feedback with OASIS Working Group
Gold release: OASIS V becomes operational under the oasis.ac.uk domain
All existing users are moved over to OASIS V
By March 2021
OASIS IV will have been decomissioned when all users migrated OASIS IV data will have been archived as a snapshot
To aid this, we have set up a number of Working Groups from across all parts of the sector to help us answer any outstanding issues.
Working Group 1: Community Groups
Focus on workflows of community groups and others working outside of development control
Working Group 2: Contractors
Focus on the workflows of archaeological contracting organisations, inlcuding those working in geophysics and maritime
Working Group 3: ALGAO (HERs and Planning+Leg)
Focus on HERs, and use within Local Authority work with development control
Working Group 4: Museums and Archives
Focus on the new ‘Level 3 user’, which allows Museums and Archives (including digital) to be a major part of project workflow, and to help signpost the requirements of collecting organisations
Working Group 5: Buildings
Focus on the use of OASIS by Built Heritage professionals and researchers. In England this will be through the use of a specific OASIS Buildings ‘skin’ tailored for the community.
Working Group 6: End-to-End Workflows
Providing an overview of how the entire OASIS system works in England and Scotland, from creation of record, re-use and engagement of stakeholders, and access to information.
Working Group 7: Historic England Internal Use
Providing a focus on HE internal teams creating records, but also as the national body for England and the insight that OASIS can provide to understanding trends in fieldwork and research
Working Group 8: Discovery and Excavation Scotland
In Scotland, OASIS feeds into the annual report of DES. This requires specific workflows and techincal requirements outside of the main form.
Working Group 9: Techincal Interoperability
To focus on the terminologies being used, howe they are being used, and how data stored in OASIS can be re-used.
Working Group 10: Training + Promotion
To coordinate and deliver a range of training and promotional activities in both England and Scotland, and to investigate new methods of learning in the digital age
It’s hoped that these Groups can begin to form the basis of a community to support OASIS and its users as things move forward. If you’re interested in learning more about a particualr group, and even joining, please contact the project email address at email@example.com
Historic Environment Scotland are holding a one day event ‘Doing more with data’ on Monday 28th October at John Sinclair House, 16 Bernard Terrace, Edinburgh. Supported by The Royal Society of Edinburgh, the event explores how we can make data work more efficiently in a collaborative environment.
Come along to hear how redevelopment of the OASIS form is progressing and our plans for release in Spring 2020. Redeveloped in partnership with Archaeology Scotland, the new form brings together the existing Discovery and Excavation in Scotland and OASIS forms into a single application. Amongst many new features, a new location module will greatly improve the quality of geographic information reported through the form and encourage users to upload their own boundary files straight into the map. Read more about that work here.
Find out about how the regional research frameworks are progressing in Scotland and discover how reporting your fieldwork through OASIS can help update the questions posed by the research framework panels. A short demonstration will demonstrate the use of the Research Frameworks module within OASIS (see below for a screen grab of current development version), which will allow users to pick from a geographically relevant list of questions their project informs. In turn, this allows the relevant Research frameworks panel to see which new projects are helping contribute to informing their questions.
The afternoon will explore the opportunities offered by digital data, challenging archaeologists to so much more with the digital data they create beyond a site plan fossilised in a pdf project report. Following an update from HES archives, further presentations will explore novel uses of data including how machine learning is transforming how we work. The House Age Project transforms scanned copies of historic OS maps into reusable vector maps. Read more about the project here. The afternoon ends with an update on working with Airborne Laser Scanning data and the potential of automated feature recognition.
Please register by Monday 21st October through Eventbrite.
One of the tasks we’re currently tackling in the OASIS redevelopment is the Location module. This is probably the most important facet of new OASIS; not only is accurate recording of a place a fundamental to everyone’s work, but within workflows project locale subsequently helps generate a lot of data in other modules of the form.
Jo has been hard at work knitting together the OpenLayers (v5) interface within a Java framework, and getting it to connect with the back-end (the classic QGIS – PostGIS stack). Without wanting to jinx things, the results are looking really good. A user can record multiple ‘sites’ within a single record, and each site can be represented by points and/or polygons. A user can also upload their own boundary files straight into the map! Once saved, a spatial query is run against an array of background mapping to generate admin areas, HER areas and (currently at review) collecting areas for Museums and Archives. Basically this tells a user where they are, and who the curators are for the record. It also allows a curator to log on and see records for ‘their’ area.
A little task we’re running in the background is coming up with ways to deal with legacy data migrated from the current database. One facet of this has been Polygons. The current OASIS database (Oracle) is not a true spatial database, and thus ‘geometries’ are stored as a set of points, but not a real polygon (if that makes sense). In order to have these polygons appear in new OASIS (PostGIS) as polygons, and not just a series of points we’ve had to do a bit of cleaning post-migration. Fortunately, we’ve been luck enough to have a fantastic Masters student (Kirsty Ackland) with us over the last few weeks. Kirsty has put my rather rusty GIS skills to shame…. In her own words:
“As part of setting up the map interface for the OASIS system an existing dataset was used to test the visualisation process. The original dataset (Oracle) is now held in a PostGreSQL/PostGIS database which was imported into QGIS. However the structure of the original data from Oracle meant the data could not be visualised as a geometry. To fix this, the dataset was run through QGIS’ database manager with a simple SQL query to retrieve only the data which was stored as polygons and output it in tabulated form to remove the geometry links. This was exported into Microsoft Excel to clean the data and remove unnecessary fields. This was then imported back into QGIS as a .csv file.
Because the geometry had been removed the records no longer had spatial references associated with them, so the next step was to find the original input co-ordinates for each record in the OASIS database. This was done using an SQL query to join of several tables within the OASIS database using the OASIS_ID, SITE_ID and GRIDREF_ID assigned to each project during data entry. This produced a table holding all of the IDs necessary to re-associate the co-ordinates (also held within this table) for each site with their assigned OASIS ID using a one-many relational join (one oasis/project ID to many gridref/co-ordinate IDs). The first dataset was then joined with the ID table using the OASIS_ID to add in the easting and northing points to their relevant project information. This was then quickly queried to remove any duplicate fields created by the process and was output into QGIS with its corrected geometry (fig 1).
To create bounding boxes for the sites held within OASIS for visualisation on the map, the convex hull tool within QGIS was used to join the co-ordinate points together a create separate polygons based on their siteID (fig 2). However, when this was complete it soon became apparent that some of the co-ordinate points were entered incorrectly resulting in the weird output seen in the screenshot where there are sharp lines running north-south. The screenshot also shows that some are in the correct place though, which suggests that this workflow works well as long as the data is correct. A secondary point to note is that some of the sites only relate to one co-ordinate point, but have been stored as “polygons” during data entry. This means although they exist within the dataset as point data, when converted to polygon to show their extent they get removed because shapefiles can only store one datatype at a time i.e. point or polygon. The effects of this can be seen in fig 3.
This is also an issue to bear in mind when considering the point data removed from the dataset right at the start of the process, as it is very likely some of the sites should be stored as polygons rather than points, leaving the visualised data output here incomplete.”
It’s at that point that I think we’re going to stop. We now have real functioning polygons in our new database. Some of them may be incorrect, but this is just the legacy of manual data entry (with no immediate visual control) in the current OASIS form, and which does date back to 2004. Hopefully the new system will deal with this alot more efficiently! Thanks to Kirsty for her quick and efficient help on this.
Just a quick blog to let everyone know that Tim and colleagues from the ADS will be at the CIfA conference in Leeds later this month.
There will be new information on the OASIS redevelopment project available from the ADS/Internet Archaeology stall, including some new promotional material .
Please come and say hello, we’re more than happy to answer any questions about what’s going on with the project. Tim will have a laptop, so (wifi permitting) he’ll be able to show you the current form as it stands!
Before I start a quick note: this is a very long blog that tries to condense about 18 months of discussion and internal reporting into an overview of a key workflow in OASIS. I’ve tried to keep it short, but for clarity i have to explain some background…
Although we all love OASIS (!), one of the key problems that has consistently been raised has been the delay in reports being transferred into the ADS Library. A Level 1 user (unit, community group, buildings archaeologist, academic) uploads a report, completes the form and then there’s either a delay or in the worst case the report never gets copied over at all.
This is normally caused by a backlog at what has traditionally been called the validating organisation: the HER/SMR. In our review of workflows in HERALD Phase 1 it became apparent that the problems at the validation level were caused by:
No HER provision
An HER was not actively validating OASIS records, and did not want to!
Lack of resources for an HER to validate as quickly as they’d like
Validating an OASIS record was taking a long time
In the current system – and where there is agreement with the HER provider – in some areas of England so-called proxy validation is being undertaken by Historic England.
After a review and comment from project funders (HE and HES) and representatives from ALGAO (HER), the new OASIS is taking significant steps to overcome this backlog while also doing as much as possible to ensure the accuracy of information and ensuring that HERs are able to delay transfer of reports where there is a critical need.
The first thing we’re doing is changing the terminology! Validation is now being replaced by ‘Review’. This more accurately represents the role most HERs perform, and separates it out from any formal sign-off related to the planning process.
In England, one of the features being implemented is the HER being able to select its level of engagement. A previous blog explains the differences, but basically an HER that is Lite can access, and edit their records but are formally ‘Not Reviewing‘. An OASIS that is Standard is ‘Reviewing‘. By default all Scottish HERs will be Standard users, and we’re currently surveying English HERs (via the HER annual survey) to establish their level of use. It should be noted that an English HER can change their Reviewing status at any time.
What does all that mean? Here’s a simplified overview of the workflow:
1) A Level 1 user uploads a report. The user is also presented with a preview of the metadata record to accompany the report in the ADS Library. In Scotland, users will see the DES preview. Level 1 users will be able to correct any errors they see in this preview record.
2) A Level 1 user will also be able to set an embargo period for the report (in months), which includes the capacity to set a “no release” value.
3) The Level 1 user will be presented with a series of alerts and challenges reminding them of the obligations of depositing uploading a report to OASIS under the (pre-agreed) terms and conditions of the OASIS Licence Agreement. These Terms include:
That the Depositor (i.e. the Level 1 user) us responsible for ensuring that the report does not include any personal details (in accordance with current GDPR legislation)
That any third party material is covered by a licence or agreement which permits its distribution within the final report.
That the Depositor is responsible for ensuring that the correct version of the report has been uploaded (with primary reference to reports forming part of a planning condition).
4) If enough of the OASIS form has been completed, the form then establishes the stages of Level 2 (HER/SMR) and Level 4 (national body) review appropriate to the record:
In England Level 2 Review status
In Scotland Level 2 Review status is defaulting to OASIS Standard.
In England there is no formal Level 4 Review.
In Scotland, Level 4 Review of the DES record will be undertaken by Archaeology Scotland (Level 4). In addition HES (Level 4) will subsequently review the record and add in relevant Canmore information.
In Scotland, transfer of the report into the ADS Library would only occur after all parts of Level 4 review were completed.
5) In the case of areas where HERs are not reviewing records and reports (OASIS Lite), transfer of metadata and report to the ADS Library is allowed to happen (i.e. as soon as the minimum level of information has been recorded and report uploaded), unless an embargo period has been set by the Level 1 user. The resulting report in the Library will have a very clear disclaimer as to its origins and the lack of Review at Level 2.
6) Where an HER is reviewing, it was deemed necessary to have some sort of mechanism to stop records forming large indefinite backlogs at HERs. This is being called the Review Period. If no action is undertaken in this period the metadata and report are automatically transferred into the ADS Library, again subject to any Level 1 embargo period, and with a very clear disclaimer as to its origins.
7)I n the case of the report being rejected, the Review Period would be reset upon the next upload by the Level 1 user.
8) Additional reports can be added to an existing OASIS record (an OASIS record never closes). Each report would have their own Review Period.
9)T here will always be the capacity for a report to be removed from the ADS Library.
The question we were asked to solve was, how long should this Review period be?
The Review Period
To this end, the question over the desired length of the period was put to ALGAO (HER) members in England and Scotland, courtesy of the relevant ALGAO representatives on the OASIS Management Board. The recommendation from Scottish members was for an initial review period of at least 12 months, with the capacity for Level 2 users to place their own embargo period (including “no release”) on any report within their mandate. The feedback from England was mixed, with a range of review periods from 3 months to a year mooted, and again with some form of ability to stop or delay a report from being released.
The need to reconcile the differing requirements for a Review Period across HERS and countries has been discussed at great length. The primary concern of HER users is the possibility that a shorter review period would result in the release of reports into the ADS Library that were not formally approved by the relevant Local Authority archaeologist/Planning Adviser, and thus potentially clash with the signing off of work/developer requirements within the planning framework. Another concern is that HERs with less resources/staff capacity would not be able to review all the uploaded reports for their administrative areas within a short period of time.
In short, everyone wants reports to be made available as quickly as possible, but we do need to have a sensible Review Period and the means for an HER to extend this where there is justified need. This has been labelled as “the capacity for Level 2 users to set an Extended Review”, with the following limitations:
The Extended Review period would not be infinite.
There would be no ability to set separate periods.
HERs that are not reviewing (England) will not have access to this function.
Level 1 users will still have the option to set their own embargo period, including ‘never release’.
However, we still need to decide how long these periods should be!
Analysis of OASIS workflows
To augment the anecdotal feedback from HERs, it was thought beneficial to look at the actual data stored in OASIS to look at how long records stayed in the system. The following analysis is based on exports of data from an 18 month period over 2017/18, and are deliberately brief to aid clarity. It is acknowledged that within them are a myriad of factors (organisational, financial, and personal) which influence the timings identified which are not considered here. However at the time of writing they are at least a neutral and statistical source on how OASIS is currently used.
OASIS workflows: Scotland
The following data was exported from the current OASIS database on 1st June 2018:
Identified 490 records completed by Level 1 user in 2017
315 of which have since been validated by the HER (64% over a period of c.18 months)
The average time from one of these records being completed by Level 1, and then validated by the HER was:
52 days (mean average)
33 days (median average)
The average time for all records (validated and awaiting validation) was:
167 days (mean average)
66 days (median average)
54% of records were processed in under 3 months
9% of records were processed in 3-6 months
13% of records were processed in 6-12 months
24% of records were processed in 12 months+
A breakdown of times by anonymised Scottish HER shows a deeply polarised picture: with most HERs (and those with least records) taking over a year to process a record; conversely, areas where OASIS is used frequently process records in a relatively short period of time.
Time that a completed OASIS record then spends with (anonymous) HER in Scotland (for all records completed by level 1 user after 1st January 2017). Each HER area is split into a bar showing the percentage of records that fall within the time periods. Dotted line shows number of records for that area
OASIS workflows: England
The following data was exported from the current OASIS database on 1st June 2018:
Identified 3376 records completed by Level 1 user in 2017
2250 of which have since been signed off by the HER (66% over a period of c.18 months)
The average time from one of these records being completed by Level 1 and signed off by the HER was:
76 days (mean average)
31 days (median average)
The average time for all records (validated and awaiting validation) was:
173 days (mean average)
115 days (median average)
47% of records were processed in under 3 months
10% of records were processed in 3-6 months
25% of records were processed in 6-12 months
18% of records were processed in 12 months+
These broad figures only aggregate out a great deal of nuance as it is well known that validation workflows within the current OASIS system vary across the country, as demonstrated by a breakdown of the 2017 data by HER (see below). A brief overview of these figures shows some clear trends:
HER working patterns currently vary significantly
The majority of areas process the majority (57%) of their records in 6 months or less.
82 % of records are processed in under 12 months.
There are some areas where validation is significantly slower.
Even areas with high numbers of records and throughput (what we may tentatively call typical users) still have a minority of records which take over 6 months to process, and in certain cases over 12 months.
Reaching a decision
The brief overview of current statistics indicates that:
Where OASIS is embedded within an HER’s workflow – and where resources permit – the time to successfully validate/review a record is below 3 months for over half of records.
In each country, the majority of records are validated in under 6 months (57% in England, 63% in Scotland)
In each country, over three quarters of records are processed in under 12 months (82% in England, 76% for Scotland)
Some records, even in areas with very efficient reviewing times, will take over 12 months to process
In trying to pick out the most suitable periods, it is worth highlighting that:
There is a need to establish a balance between being aspirational and being realistic.
The new system will improve the quality of recording by level 1 user.
Alerts will be in place to aim to prevent uploading of unsuitable reports. This can include guidance from each HER on how to use the form in their area/expectations etc.
The use of the embargo period seems to be well established in many areas, which could be taken to indicate that dialogues between Level 1 users and Development Control/Archaeological Advisers on what should be uploaded to OASIS – and limits to accessibility – are already in place. However it is also clear that a form of Level 2 override is still needed.
Although safety nets are planned, there are obviously still cases where a Level 2 ‘override’ is needed.
In Scotland the workflow through DES-HES will still be in place.
Both the initial Review and Extended Review period could be changed at any point.
At this stage, it is suggested that as the majority of records from each country are successfully reviewed/validated in under 6 months that an initial Review period of 6 months is set as the starting value in the new OASIS system. As there seems to be a significant number of records which do take up to a year to process, it is suggested that there is a strong need for an Extended Review button/option being able to add on an additional 6 months when triggered, meaning a record would have a cumulative total of 12 months under review in the first instance. If rejected by the Level 2 user, the initial Review period will still be in place upon upload of a new report, giving the Level 2 user a further 6 months to act.
Overview of the Review / Extended Review workflow
The option for Extended Review will appear in the Report section of the form, and will also include an additional field for the Level 2 user to record a description of why the report is being withheld if they wish. This will be available to anyone with access to that record.
To allow the Level 2 user to have as clear a picture as possible, the following alerts and information displays are being built into the new OASIS form.
Current release date of report will appear on overview e.g. “Report will automatically be released into ADS Library on 01/04/2030 if no action taken”
Level 2 user will be able to filter records on Release date to see imminent releases
Level 2 user can set their email preferences to send a digest of reports nearing end of their Review period, e.g. “The following reports will be released into ADS Library next month:
The Extended Review period will automatically add 6 months onto the existing Review period, this is best demonstrated with some hypothetical examples:
Level 1 user uploads report a report to OASIS on 1st January 2030. All relevant sections of the form are complete, and record can pass to Level 2 review (OASIS Standard). No action is taken by Level 2. The report is released into the ADS Library on 1st July 2030 by default (with disclaimer)
Level 1 user uploads report to OASIS on 1st January 2030. All relevant sections of the form are complete, and record can pass to Level 2 review (OASIS Standard). Level 2 user reviews record in 1st April 2030, no problems and report goes into the ADS Library.
Level 1 user uploads report a report to OASIS on 1st January 2030. All relevant sections of the form are complete, and record can pass to Level 2 review (OASIS Standard). Level 2 user reviews record in 1st April 2030, has a query over the version/contents of the report, clicks “Query/refuse report” and communicates this back to Level 1 user with a request that report is re-submitted. The record is marked as ‘awaiting upload of report’. On 1st June 2030 the Level 1 user uploads a new version of the report. The Review Period is re-set to 6 months from this point, meaning that if the HER does not perform any action the report automatically goes into the Library on the 1st December 2030.
Level 1 user uploads report a report to OASIS on 1st January 2030. All relevant sections of the form are complete, and record can pass to Level 2 review (OASIS Standard). On the 1st June 2030 the Level 2 user is alerted to the fact that report will be released next month until action is taken, Level 2 user is busy and wishes to give themselves more time to check contents and status with DC, and clicks ‘Extend Review’. The report is now due to be released on 1st January 2031 (i.e. 12 months after report was initially uploaded).
As above, on 1st November Level 2 user reviews record and identifies problem with the report, clicks “Query/refuse report” and communicates this back to Level 1 user with a request that report is re-submitted. The record is marked as ‘awaiting upload of report’. On 1st January 2031 Level 1 user uploads a new copy of the report with another 6 month review period if no action is taken by Level 2. The report is released into the ADS Library on 1st July 2031 by default (with disclaimer). The option to click another Extended Review is now not available. If the Level 2 user is still not happy with the contents of the report they have this additional 6 months to refuse it, effectively resetting the Review Period each time.
That’s the plan! This has been presented to English and Scottish HERs and there is agreement that this is a sensible way to proceed. This will be reviewed, tro see if the windows are working as intended, and guidance/training on this workflow for all users will be a key feature of the public rollout of the system. However, time to stop writing!
In England the ADS (via the Historic England HIPs team) are currently asking HERs about their preferred level of engagement with OASIS. This is because in the new OASIS system we now have three tiers of recording instead of just one: OASIS Lite, OASIS Standard and OASIS Plus.
[Note: that in Scotland, because of the the workflow designed to communicate results to HERs, HES and DES, OASIS Lite will not be an option.
Another note: OASIS Plus are specialist modules designed to record fine detail about very specific types of events such as geophysical survey, that fall outside of the main Level 1 (data in-putter) to Level 2 (HER) workflows and thus can be put to one side for the purposes of a concise blog.]
The two main workflows – OASIS Standard and Lite – have been created to reflect the realities of the levels of HER engagement we see in the current system, and reinforced by feedback to the original HERALD phase 1 survey on what people wanted in the redesign. The fine details of these workflows, and the list of mandatory fields are best viewed in the online version of the Functional Specification
However, I thought it best to provide an ‘at a glance’ overview of the differences between the two systems, so without further ado…
The HER (Level 2) is actively participating in OASIS and wants to hold reports and metadata for review before they are available to the public.
So, the Level 1 user fills in the mandatory fields for OASIS standard and uploads a report. Once everything is completed the HER has a window to review, or highlight records are held for extended review.
When review is completed the report and metadata go into the ADS Library.
OASIS LITE is a mechanism for uploading project reports to the ADS Library.
OASIS LITE will collect an enhanced bibliographic record which is used as discovery metadata for locating the report (so less than OASIS Standard)
OASIS LITE only applies to the areas of the form which are relevant to transfer to the HER (i.e. not the archive section which will be dependent on participation by the museum/archive rather than the HER).
OASIS LITE will only be available when there is an HER for the area and that HER is collecting the full event record by other means (i.e. not via OASIS). All other OASIS modules which are relevant to that event/project will be collected even if the HER is not using OASIS i.e. Archive and Geophysics.
Any reports that are released into the ADS Library through this mechanism will have a very clear disclaimer explaining that they have not been reviewed by an HER. The text of this disclaimer is being written by ALGAO executive with input from HE/HES.
There’s doubtless a few questions, I’ve tried to preempt them below:
What are the mandatory fields for OASIS Lite and Standard?
Can an OASIS Lite have more than the mandatory fields recorded?
Yes, if the Level 1 user wants to, they can fill out as much as they want.
Why are you asking now?
As part of the package of work establishing the review and extended review (i.e. how long an HER using OASIS Standard has to review a completed record), we need to get an idea of the split between Lite and Standard users. We also need to have the participation level set for when we start testing workflows for Standard and Lite.
Can an OASIS Lite HER still log in and see records/export data etc?
Yes. They still have access to all the data for records in their area, with access to any exports and API functionality.
Can the level of participation be changed?
In short: Yes. The HER user simply clicks a button in their admin panel.
In more detail: when the record is started it is defined as Lite or Standard in the database, a separate audit table also records when a record is saved, mandatory fields present depending on level (i.e. “complete”), report transferred to Library and so on. This will be reflected in what the user sees in their list of records.
If an HER swaps from Lite to Standard there is a good chance the HER could suddenly get records to review, but only ones which hadn’t been defined as completed already and transferred to the library. This will mean however that a Level 1 user will be presented with increased requirements for any records that have been left open. However the differences in mandatory metadata requirements are not onerous, the main impact will be on the timing of transfer of reports into the ADS Library (i.e. any live or new records will be subject to HER review).
If an HER swaps from Standard to Lite then completed records will be unaffected, incomplete records will simply have less for the Level 1 to complete as mandatory and report transfer will default to no review.
That’s the plan, however we will be reviewing how this works in practice!
If I change from Lite to Standard, can I edit completed Lite records?
Yes. However you can’t request that a Level 1 user goes back and edits as they have completed what was required of them at the time. The record will still be marked as a Lite record, reflecting its original status.
We have identified a potential issue where a Level 2 user switches to Standard, and then may want to start correcting or amending completed Lite records for their area.
If an HER wants to go and add or edit that’s fine. Any OASIS record is always open, so additions or corrections are fine. The only sticking point is the report…. When a report and its metadata are transferred into the ADS Library that version is what’s archived as a point-in-time exercise. If an edit to report metadata is required it can be made by either the Level 1 user or HER responsible for that record directly in the ADS Library (we have created a module to do this!). This will need care, but basically there will be a system where accredited users can update Library records themselves without making a request to the ADS.
Hopefully because of the new functionalities in the data entry form there should be many fewer errors in the basic bibliographic record, but we have to be prepared!
Things get trickier if the HER, once switched to Standard, notices problems with the completed Lite reports uploaded for their record. We have to try and balance the integrity of what was originally recorded (according to what was required, and the disclaimer) with the need to add or remove new reports where urgently needed. In this case there are options:
The HER simply uploads the correct version of the report, both versions of the report are online (the first with a disclaimer). Potential to have links between the two records so that a user finding the first, is aware of the second.
The HER finds the report they want to remove in the ADS Library, and flags it for removal/hiding. The ADS are then notified and the report is removed. Level 1 users will also have the ability to flag reports for removal (please note that if a report is deleted, the DOI remains, but just points to a landing page notifying the user that it has been removed).
It should be noted that the above are options, and nothing is completely set in stone. There’s a a balance between creating a system that allows the upload of reports in a timely manner and the ability to fix genuine mistakes or remove sensitive information, with the risk of having a system that becomes a mechanism for a constant cycle of upload-removal-upload-removal of different versions of the same report by different people.
We will be exploring this further with all relevant parties during the ongoing development, but as always I’m happy for thoughts/ideas/concerns to be raised, so feel free to email firstname.lastname@example.org
Over the last few weeks (ether side of Christmas) we’ve been making some progress on the part of the new OASIS which records the archive. As an archival body ourselves we’re keen – along with everyone else I’ve spoken to – that the new system improves on:
Recording what has been found/produced for archive
Allowing an archival body to produce in-form guidance on what it expects from a deposition
Making the archival body aware of events happening within their area/remit
Allowing the archival body and data producer to correspond at an early stage
Recording the deposition stage
Reflecting the differences in archive workflows in England + Scotland.
Signposting between physical and digital archives
One of the problems with the old OASIS was that records were often completed whilst the archive was still in flux. Looking over many records you can see cases where archive location is “TBC” or “Intending to send to…”. There’s also a inconsistency in naming the organisation, and a great deal of uncertainty in what people are recording is in the archive. It’s not all doom and gloom, there are some really good examples (everything neatly recorded, archive named, museum accession code in place) that show that the willingness to record the archive properly is in place among many archive users.
The first thing we can do to help is to have an OASIS form permanently open. So if the allocation and deposition of archive is on a separate timeline to the rest of the record (as often seems to be the case), then it’s a simple matter to have a user return and complete when things are sorted. That’s especially true in Scotland where (at the time of writing) the allocation of archives is decided via the Scottish Finds Allocation Panel as Treasure Trove – although people may have an idea of where things may be deposited based on past examples, the archive part of OASIS needs to stay live so that it can be updated where needed.
The other thing we’re doing is to have the curators involved (as much as possible) from the outset – archivists can create their own OASIS accounts, create their organisations, upload guidance on workflows, attach pro-forma and so on. This is a big step for OASIS to take. Historically OASIS has just been used for data producers, HERs and national bodies (a consequence of its origins as a simple event recording system). And while over the years we have created bespoke views for particular data consumers (where appropriate) it’s clear that for OASIS to truly succeed in it loftier aims for tying together all parts of the Historic Environment, it really has to get archivists engaged with the system.
So what functionality are we looking at introducing?
Introduce workflows to incorporate what we’re calling Level 3 users (museums + archives)
Ability for Level 3 to create guidance within the OASIS form on how the workflow operates in their area, if they’re accepting archives, sorts of archive they accept, online deposition forms etc
The ability for a Level 3 user to set their geographic collecting area. This may be county based, regional even national.
The ability for a Level 3 user to see things that may be coming or should be coming to them (based primarily on location)
The ability for a Level 1 user (unit, academic, community group) to see the archives collecting in their area, and to check up on what’s expected
The ability (via a section called archive notes) for a Level 3 user to correspond with a Level 1 user. This records any back and forth in the form allowing people to track what’s going on (see below)
An example of the OASIS form recording correspondence between Level 1 + Level 3 users about archive guidance. Please note, this is not definitive but just to give an impression.
The actual part of the form where archive contents are recorded is (hopefully) straightforward (see below for an impression). In this a Level 1 or Level 3 user can record:
Where an archive is prior to deposition
In Scotland, when the archive was sent to SFAP, and the decision of the Panel as to appropriate course of action.
Where the archive was deposited
The accession/archive ID
The expected deposition date
The date it was deposited
If the archive was accepted or refused (including date)
There’s also space for a contents list of pro-forma template if required by the archive.
A mock-up of basic functionality in the main archive page. Please note, this is not definitive but just to give an impression.
And finally, what’s actually in the archive! This has been tricky…. In a review of the current OASIS system by FISH, the archive contents page and internal lists was noted as being a touch confusing, and understandably out of date. The recommendation for the new system is that as much as possible existing heritage thesauri are used. In-particular that the user can select:
However that still leaves the written, drawn and digital side… An initial list called ‘Paper and Digital Archive Component’ recording basic concepts such as, stratigraphic matrix, section drawing, photogrammetric model etc has been established courtesy of the Historic England’s DSU and reviewed by FISH. The list – to begin with – is deliberately simple, and there’s still plenty of time to review and propose additions to the list. As ever with a development project on a relatively short timeline – we want to get the basics working first.
The final piece of work has been to try and compile a list of archives, record offices and museums in England and Scotland. In order to:
Map multiple terms used in the old OASIS to a single known entity
Have a list of organisations + data ready to use in the next phase of testing.
To begin with we’ve compiled a list of museums maintained by the SMA (see https://doi.org/10.5284/1018089) along with the list of organisations listed by The National Archives. For each one – and where available – we’ve used details such as location, whether they’re collecting, website, collecting area and so on. Although very rough, this is enough to start planning and then testing these specific Level 3 workflows.
As ever, I’m very keen for specialists from this sector to be involved in reviewing what we’ve doing (in terms of the lists we’re drawing up), and to volunteer to test and feedback on the workflows. So if you are reading this, and happy to help then do please get in touch at the usual email address: email@example.com
Following on from the recent blog on the behind-the-scenes work with vocabularies and migration, here’s a short overview of our work on developing the form itself…
One of the most important features of the new OASIS system will be a much enhanced system for logging in. This sounds incredibly straightforward, everyone is used to a web-based system that asks for a username and password; but behind OASIS is a level of technical complexity to support an increased need for simplicity and flexibility. This sounds somewhat contradictory, but to explain…
In the current system logging in is done as an organisation – all users within this organisation (for example My Archaeological Unit) log on with the same details. But what about where there are lots of people, and what if these people are split between regional offices, and what happens when one person forgets the password? What happens when someone leaves? Do we change the password, or rely on the individual to forget it (admittedly I’m sure everyone has much better things to do than logon and sabotage OASIS records, but the principle remains)?
The solution is to move to a system where every individual has their own logon credentials, but then the real complexity starts as we have to factor an individual moving jobs, having more than one job/role and for these roles to have different administration rights. For example, in a hypothetical situation I may start off working for Anfield Archaeology in a role that involves me creating and editing OASIS records for my projects. Then I move to Goodison Archaeology and take over pre-existing projects: the system needs a process to stop me seeing Anfield records (commercial sensitivity etc), but to have access to Goodison records. Whilst in my new job I do some work with Spellow Archaeological Society and offer to help them fill in some archaeological records, as an experienced OASIS user I volunteer to set up this group in the system and become an admin user, but at a later data want to pass on this responsibility to someone else.
I’ll stop there, but the the myriad of potential roles and responsibilities – and these are things identified by OASIS users in the survey – are enormous. So now we’re no longer just dealing with a log on page, but a full-blown user management system. This is what Jo’s been hard at work on over the Summer. Explaining what she’s done in prose is overly long, so here’s a few screen grabs to demonstrate the main principles. Please remember this is the first draft version of the form, things aren’t set in stone and these grabs should be used for illustrative purposes only.
Slide 1: The new OASIS homepage (login on top right corner of screen)
Slide 2: Presuming I already have an account (I get to registering below), I logon and am met with a landing screen. The top-right corner tells me who I’m logged on as, and which organisation I’m currently “in”. In this example, I’ve set my own fictional unit Tim Evans Archaeology as being my default organisation. I can select an option to look at my personal profile
Slide 3: Here I can change my name, I can also see that I’m a standard user for the organisation (so no admin rights). Further down the screen I can set the frequency of notifications I receive from OASIS.
Slide 4: If I go back to my home-page, I can see that I’m also registered with two other organisations. In this case I want to go to my work with “HERALD Level 1 Test User”.
Slide 5: I’m now “in” the system as this other organisation, I can see which organisation I am by the notification in the top-right corner of the screen. If I look at the organisation profile I can see the list of users registered to this organisation. I can see and edit this section this as I am a registered ADMIN. In this example you can see that the user timevans1878 has registered for my organisation, but their user status is set as PENDING. People asking to join an organisation and see their records have to be vetted by an ADMIN, which in this case can be any of the 3 users with these privileges (note: there can be as many admins as needed).
Slide 6: If I scroll down that a page, I can see some extra functions. At the bottom there’s the option to add some text and a logo for my organisation. This will appear in OASIS (so can be seen by other users), and also be transferred to the ADS Library to accompany my organisations reports. Branding of reports (and the ability to edit logos) was a popular request in the scoping phase and is perfectly understandable; people put a lot of work in getting their reports online, and it’s good to enable them to customise “their” section of the site (for example see Wessex Archaeology in the ADS Library). The other function to flag up is the add person button (highlighted)
Slide 7: If I click this I come to a screen where I can start searching for other registered users, I start typing in for my colleague Julian Richards and I can see that he’s in the system. I select him and can either add him as a Standard or Admin user. Important to remember that I can do this due to already being an ADMIN. I can downgrade my status at any point, although the system will always require 1 ADMIN user.
Slide 8: I can also leave an organisation if I wish, so simply go back to my profile and I can see that I’m still registered with “Tim Evans Archaeology” and “test Katie”, but can leave as long as I’m not the sole admin. You can also see that I can apply to join other organisations.
Slide 9: this all assumes I’m already registered. If I’m new to OASIS I can apply to register instead of logon. In this slide I’ve just been asked to register using my email address. However having done so I’m reminded that tim.evans is already registered as an OASIS user, I then get the usual reminders/reset. This is to try and stop (as much as possible) duplication within the system. An interesting facet of this work is our aim to coordinate OASIS users logon details across all main public facing applications hosted by by the ADS (OASIS, Library, ADS-EASY). Jo has been hard at work wrestling with things such as composite persistence modules (no, me neither) to reach a point where a user just has one set of ‘centralised’ credentials. So, no more logging in/out of different systems.
Slide 11: My fictional user is then asked to which organisation they would like to be a part of. Here, the user wants to be part of Tim Evans Archaeology. They find the group then click to join, as noted earlier their participation will need to be approved by an ADMIN for that group. If my organisation doesn’t exist, I can create my own
Slide 12: So off I go and create my new organisation. The system does check to see if “Steve Puffin Archaeology” already exists to try and stop duplication. Providing it doesn’t, the new account is created and I’m automatically the ADMIN for this group. The world of OASIS is now my oyster.
I’ll stop there! I hope that’s given people a clear overview of just some of the functionality that’s already been built. We’re currently only sharing this pre-alpha build with the project funders, but by Spring 2019 we’ll be circulating the next version of the build for wider testing. We’ve already had several people sign up, so if you are interested in being an early stage tester please do get in touch via firstname.lastname@example.org
As the nights begin to draw in, it seems like an opportune moment to reflect on the progress of the redevelopment of OASIS. The Summer has perhaps been atypical here, usually it’s something of a hiatus between the rushes of finishing things to meet the end of a financial year/going on holiday, and beginning afresh in the Autumn. Not so this Summer, and it seems like myself and Jo have been busier than ever, but what have we been doing?
The first part of this blog is to try to give an overview of the non-technical tasks, I’ll be writing up the development team’s work on the actual form as a second part shortly.
Hopefully readers will have seen the alert for the small user needs Survey in September. This has now closed and the results are being written up to feed back to HE and HES on the resources that will be needed as we move into Stage 3 of the project (the actual roll-out of a real, live system). There was some valuable feedback, particularly on the value many users placed on having a human presence to help with their queries, greater input into training/guidance from people that actually use the form in day-to- day workflows (so not me!), and to ensure support for users in remoter parts of the country. I’ll write a longer blog on the results in the near future.
Elsewhere, we’ve also completed a task to ensure that the new system incorporates the highest standards of recording. As users of the current form are no doubt aware, there have long been problems with ensuring the accuracy and consistency of things being recorded in OASIS (a consequence of the age and original purpose of the form, but that’s another story). During the first stages of the redevelopment project exactly whatis being recorded and howit is being recorded, came under scrutiny. The result is that some fields have been dropped, others merged or rationalised. A full list of what remains can be seen in the Functional Specification which is available on the HERALD project wiki for those that wish to find out more.
Moreover, there’s the challenge of ensuring what goes in these fields is accurate and understood. So at long last we’ll be using the full power of the thesauri hosted at heritagedata to enable users to enter correct terminologies. And yes, they will be country specific where necessary. The form is being built to be flexible enough to have a drop-down for simple lists, but to allow users to enter their own terms and return relevant entries from the thesaurus (see below)
For new OASIS we’ve wanted to go further and to try to use wordlists wherever possible. This has led to a mini project to identify areas of the form that could be controlled (so no free text), and then create the list of terms to be used. Since an initial review by FISH back in December 2017, this work has been aided by the teams at HE, HES and Department of Environment for Northern Ireland, especially Zhouyi Qian a placement with HE’s Data Standards Unit. In a short space of time Zhouyi was able to rationalise 16 years worth of entries to OASIS into smaller groups of terms to describe:
The reason for the investigation e.g. <Planning requirement> or <Ecclesiastical Consent >
The broad development type (if related to an event within the planning framework) e.g. <Land management> or <Energy and power generation>
The broad funder type e.g. <Utilities and infrastructure> or <Charitable organization>
Site protection status e.g. <Designated Wreck> or <Guardianship Monument>
Types of site identifier e.g. <Canmore Event ID>
Paper + Digital Archive component e.g. <Aerial photograph transcription>
After a review by FISH members, the lists are good to go. As if that wasn’t enough, Zhouyi undertook the almost Herculean task of mapping all entries within the current database to those new concepts. To give an example of the extent of this task, consider the values entered to record the old fields for Prompt and Reason for Investigation (merged into Reason for Investigation). In the old system there were 69,750 values, in 1376 unique combinations such as:
Direction from Local Planning Authority – PPG16
Direction from Local Planning Authority – NPPG18
Direction from Local Planning Authority – Direction 4
Planning agreement (Section 106 or 52)
Direction from Local Planning Authority – PAN42 Article 4 Direction
and so on
Alot of these were pretty straightforward, so for example:
“Direction from Local Planning Authority – PPS” = <Planning requirement>
“Direction from Local Planning Authority – PPG16” = <Planning requirement>
Others were somewhat opaque, for example:
“To inform reconstruction”
“I was told to do this”
and my favourite: “The wall fell down”.
It’s important to note that all the original data will be kept (nothing is being deleted!!!). So for example you’ll be able to see that for example, in the original form a user recorded “Everton Football Club” as the funder but that this has been mapped to the concept of a <Private or public corporation>. We will also be archiving the complete OASIS database in perpetuity.
As well as the new lists we’ve also been mapping and migrating standard relevant parts of the form to the correct entries in the more well-established thesauri (monument, event, object, period). As these have been free-text there is a great deal of variety in what and how things have been entered, but my favourite is the person who for object recorded “WOTSITS PACKET”. This valuable mapping work is allowing us to migrate records from the old to new database, to help populate the new form and so that everyone’s records will appear when they finally get to logon!
Speaking of which, time to move on to part II and show off the new form…