Introducing the OASIS API

Model of a Connecting Rod – Rijksmuseum, Netherlands – Public Domain.
https://www.europeana.eu/en/item/90402/NG_MC_1889_50

In March 2022 we quietly launched the first version of the OASIS REST API, which provides access to information about OASIS Projects via a RESTful API.

It is designed to provide programmatic access to OASIS Project information and its output is very similar to that of the existing “Export” function that can be found on the OASIS Projects list. Links to the API and documentation are below.

https://api.oasis.ac.uk/

https://docs.oasis.ac.uk/api/

The API is available to all registered OASIS users, and gives you access to the records you can normally acccess in the system. So for example, an HER has access to all their HER records, a Unit has access to all their records and so on.

This is new territory for OASIS, but something that was specified as essential for increasing machine interoperability of OASIS metadata. We hope to be working with some parterns to identify case studies of how and why the API has been used, and publicise these more widely. In the interim, if you are interested in using the API have a look at the documentation page linked above. Any problems or questions, then please contact the OASIS Helpdesk oasis@oasis

OASIS V: recent updates and new features

A craftsman working at his bench. Woodcut. – Wellcome Collection, United Kingdom – CC BY.
https://www.europeana.eu/en/item/9200579/tt6xkhmg

Since the launch of the new system the OASIS Helpdesk has received plenty of feedback from users. This has been a mixture of pointing out small bugs or glitches, or else requests for tweaks or additional features to help people with their workflows.

The most recent version of OASIS (deployed 16th May 2022) inlcudes a number of updates we hope that all users will find helpful. These include:

1 More features on the Search page

a)Ability to search on: Museum Accession ID, HER Event ID, HER monument number, Site code, Project Identifier.

b) Ability for HERs to search by review date, to help prioritise records that require more immediate attention.

c) Ability to filter out just records awaiting transfer to the ADS Library, or just those in the ADS Library.

d) Ability to filter on event type.

2 Archive messages bug

The link to this feature had a bug, which has now been fixed. You can now contact your local Archive or Museum whenever you want!

3 Notification email link bug

The link to the user profile section that was creating a 500 error has been fixed.

4 Cancelling on Activity page bug

A strange bug on the Activity page has been fixed. This was where where a user clicked cancel and the page either gave an error or saved the values within that session. This button should now clear any changes made in that session, return to the previous values entered, and not generate an error. This new code has been replicated on most other pages to deal with any other instances of ‘cancel’ causing issues.

5 Watched projects export with no selection causes error bug

This has been resolved, a user is now informed that they have not selected any projects to export.

6 Ability to batch unwatch

On the watched projects page, there is now a button at the bottom of the project list that allows you to batch unwatch seleccted projects.

7 Improvements to notifications + watches

Some major improvements have been made to the notifications: automatic notifications should now be sent for:

  • Reports transferred to ADS Library (with DOI link)
  • Digital Archive transferred to the ADS (with DOI link)

In addition, there has been some re-working of the “watch” function that determined who receives updates. Watches are now removed 40 days after the report has been transferred to the ADS library, or if no report exists 40 days after the review. However, the watch is re-established if additional reports are uploaded, or extra information is entered into the archives section. This is an attempt to keep people in the loop where they would need to know about important extra information, but not small things like typos.

8 Improvements to report page

Users have reported some problems with the status of the report page not reflecting a most recent action: for example the deletion of a report not resetting the core fields to incomplete. These were rare, but frustrating cases often caused by the form becoming out of sync with user actions. The code has been re-worked, and this page should be working much better!

9 File uploads (archives and reviewers page)

Users on the Archive page and the Reviewers Information page should now be able to upload a wider variuety of files: PDF, DOC, DOCX, XLS, XLSX

10 Results page

Frequent users will have already spotted that this has changed slightly over the last few months, with users asked if their project contributes to a Research Framework. Some extra work has gone into the page to better guide users on how to fill in if they answer “yes”. We’re still working with the relevant parties from the Research Frameworks to populate the questions.

11 Organisation details page

The ‘malformed XML’ alerts when users updated details has been fixed.

Say Goodbye to OASIS Images!

As many readers will be aware the a new OASIS system (OASIS V) is now in place. In preparation for this we have taken the decision to remove the OASIS Images function from the current OASIS IV system from the 1st of April 2021.

What! Will this make depositing more expensive I hear you say?

The simple answer is no! In fact we will be reducing our standard ADS-easy set up fee from £200.00 to £150.00 for all ADS-easy deposited archives so archiving will generally become cheaper.

For all archives submitted via ADS-easy from the 1st of April 2021 a set fee of £150.00 (exclusive of VAT) will apply. As as part of this set fee depositors will be able to deposit up to 150 jpg/tiff images at no extra cost. Additional files will then be charged on a per file basis according to our current per file charges.

How will I deposit my photos from 1st of April 2021?

From the 1st of April 2021 depositors just need to log into the ADS-easy system without having to follow the more complex OASIS Image login process from OASIS IV. OASIS IV will include a notification to let users know about the change.

Why now?

Continue reading Say Goodbye to OASIS Images!

OASIS V is here

Image of the painting Basque Celebration (dance at El Antiguo, San Sebastián) . Santiago Espona Bequest, 1958
Basque Celebration (dance at El Antiguo, San Sebastián) . Santiago Espona Bequest, 1958. CC BY-NC-ND

We’re happy to annouce that after much work, with the support of funders Historic England and Historic Environment Scotland, we have a new OASIS system coming online!

This is the fifth iteration of the OASIS form; it began life as a pilot/concept in 2001 and was last updated in 2007 when OASIS was expanded to cover Scotland and a further module
to record fine detail of geophysical surveys (OASIS IV). The latest redevelopment introduces a myriad of improvements and features requested by all the user community and which we sincerely hope will make it easier to use. As a component of HIAS in England, Scotland’s Archaeology Strategy, and facilitating reporting to Archaeology Scotland’s Discovery and Excavation in Scotland, OASIS will also play a pivotal role in facilitating the sharing and access of information from all parts of the sector.

Since its formal rollout in England in Spring 2004 OASIS has collected the
details of over 90,000 records from England, Scotland, and the maritime region of Wales, and made the unpublished reports of 50,000 of these freely available in the Library of the ADS. It’s also an intensively used system, with over 1400 registered organisations, and hundreds of edits and new records made every week. Managing the move over to the new system is thus an intensive and careful piece of work. As OASIS V is also very new, users will also need to be supported with as much Training and Guidance as possible.

What’s Happening Now?

Image of a Fresco depicting Janus - Roman god of transition
Fresco of Janus: the the Roman god of beginnings and transitions
Fresko N.N., Foto Maurizio Fabre, Public domain, via Wikimedia Commons

As of ‘now’ when you go to the OASIS website the landing page will look different! There is a link to the current system users are familiar with, and the brand new one (OASIS V).

In England we’re beginning to move Level 1 (Researchers, by which we mean anyone that creates records) and Level 2 (HERs) users across to the new system. We’re doing this in an orderly, piecemeal fashion. All Level 1 and Level 2 users should have been emailed with instructions on what to do to verify their accounts, and further information relevant to their role and organisation. If you’re reading this, and your organisation has not received an email then please contact herald@ads.ac.uk. It’s important to say that please do not try and login to OASIS V until you’ve heard directly from us that your account is in place, users verified, records migrated. In the interim, please continue to use OASIS IV as normal.

In Scotland, we’re looking at a slightly different timescale. We’re still due to finish a few Scottish-specific features that facilitate reporting to Archaeology Scotland’s Discovery and Excavation in Scotland. Level 1 and Level 2 users are being emailed as well in due course, but we expect rollout to begin in the New Year (2021) and with a suite of training materials specifically for the Scottish workflow (see below). As with England, please do not try and login to OASIS V until you’ve heard directly from us that your account is in place, users verified, records migrated. In the interim, please continue to use OASIS IV as normal.

An important part of OASIS V is linking to documentary, physical and digital archives. As such, a completely new feature is repositories themselves being added into OASIS as what we’re calling a Level 3 Reviewer. This enables researchers to see which archives cover their project, and what the status and requirements of the repository is. Conversely, it will allow Museums and Archives staff to see “what’s coming their way”, and to be able to lead on ensuring requirements are followed. As this is completely new, and that we’re aware most Musuems and Archives staff have had little historic involvement, we’re spending a bit more time getting these users set up. We’ve already contacted a few users, and are sending out alerts via colleagues in SMA. However, if you work for a Museum and Archive and are reading this and haven’t received any other information but would like to know more, then get in touch with herald@ads.ac.uk and we’ll be happy to help.

What Help, Training and Support is available?

There is Help Text throughout the form, and a designated Help page which links out to a new Manual. There are also a number of Videos to guide you through particular workflows and technologies. These are being added to all the time, and are available on a designated OASIS Training Video Channel. Users of these will note that they’re a mix of narrated and unnarrated – this is deliberate and a consequence of the frequent lack of quiet space over lockdown!

For those working in England, there’s a series of online workshops being organised by MSDS Marine in partnership with Ashtree Heritage. You’ll also see more details being added on the OASIS blog, and we’ll do as much as possible to publicise further events in the future. As noted above there’ll be Scottish specific training arriving in the New Year.

As iterated above, if you have any problems or queries about this process then please get in touch with us at herald@ads.ac.uk. This is the OASIS redevelopment project account so emailing that will mean your query gets escalated and resolved as soon as possible

OASIS V Beta and timeline for development

A reading woman tries in vain to stop Chronos as he passes by. Stipple engraving. Credit: Wellcome CollectionAttribution 4.0 International (CC BY 4.0)

Progress update

Despite the current tumutlous events, the redevelopment of the new OASIS form has continued. In March 2020, the ‘Beta’ system was demonstrated (remotely) to the OASIS Management Board for comment. Although a few areas still need finessing (as with any IT development), the core infrastructure has been completed. We are now beginning the transition from a Beta (effectively a working draft in this case) to a fully live system that replaces the current form.

In the various information you may have seen, the new version of OASIS is sometimes synonymous with HERALD (a project name). However, we have always been sticking with the name OASIS for the system itself. The current version everyone is using now (as I write in April 2020) is the fourth major iteration deployed since 2002. Internally, we often refer to it as OASIS IV. It is important to note that throughout all subsequent discussion we’re simply going to refer to the new version as OASIS V.

In many ways. moving between versions is more complex than ‘just’ code development: OASIS has over 88,000 records , and over 5000 unique users representing 1386 organisations. It’s also used intensively.

  • In 2019 there were 108,000+ requests for pages within the OASIS form. We don’t know what these are though (could be one person clicking through alot of times!).
  • In 2019 there were 19919 edits to forms that resulted in a new version

We simply can’t suddenly move everyone over in a week! We need to gauge performance, iron out some issues, and start familiarising as many people as we can with how the new system differs from the old one. Most importantly, we don’t want a user to be inputting data into two systems and duplicating effort. This is the next phase: “Rollout of OASIS V”.

Timeline for Rollout of OASIS V

The following is the planned timeline for the next phase. It is important to note, that unless directed otherwise over this period, use of OASIS IV (the current form) should continue until the Autumn of 2020.

April 2020HER familiarisation with OASIS V
Internal testing
Review of OASIS V stability under increased use
May 2020Development Sprint
Testing and Feedback with OASIS V Working Group
June 2020Silver release of OASIS V: a major redeployment
Moving a small number of early adopters over to OASIS V
July 2020Development Sprint
Testing and Feedback with OASIS Working Group
August 2020Development Sprint
Testing and Feedback with OASIS Working Group
September 2020Gold release: OASIS V becomes operational under the oasis.ac.uk domain
October 2020+All existing users are moved over to OASIS V
By March 2021OASIS IV will have been decomissioned when all users migrated
OASIS IV data will have been archived as a snapshot

Working Groups

To aid this, we have set up a number of Working Groups from across all parts of the sector to help us answer any outstanding issues.

Working Group 1: Community GroupsFocus on workflows of community groups and others working outside of development control
Working Group 2: ContractorsFocus on the workflows of archaeological contracting organisations, inlcuding those working in geophysics and maritime
Working Group 3: ALGAO (HERs and Planning+Leg)Focus on HERs, and use within Local Authority work with development control
Working Group 4: Museums and ArchivesFocus on the new ‘Level 3 user’, which allows Museums and Archives (including digital) to be a major part of project workflow, and to help signpost the requirements of collecting organisations
Working Group 5: BuildingsFocus on the use of OASIS by Built Heritage professionals and researchers. In England this will be through the use of a specific OASIS Buildings ‘skin’ tailored for the community.
Working Group 6: End-to-End WorkflowsProviding an overview of how the entire OASIS system works in England and Scotland, from creation of record, re-use and engagement of stakeholders, and access to information.
Working Group 7: Historic England Internal UseProviding a focus on HE internal teams creating records, but also as the national body for England and the insight that OASIS can provide to understanding trends in fieldwork and research
Working Group 8: Discovery and Excavation ScotlandIn Scotland, OASIS feeds into the annual report of DES. This requires specific workflows and techincal requirements outside of the main form.
Working Group 9: Techincal InteroperabilityTo focus on the terminologies being used, howe they are being used, and how data stored in OASIS can be re-used.
Working Group 10: Training + PromotionTo coordinate and deliver a range of training and promotional activities in both England and Scotland, and to investigate new methods of learning in the digital age

It’s hoped that these Groups can begin to form the basis of a community to support OASIS and its users as things move forward. If you’re interested in learning more about a particualr group, and even joining, please contact the project email address at herald@ads.ac.uk

Connecting OASIS and DES with Regional Research Frameworks in Scotland

Peter McKeague, Historic Environment Scotland

Historic Environment Scotland are holding a one day event ‘Doing more with data’ on Monday 28th October at John Sinclair House, 16 Bernard Terrace, Edinburgh. Supported by The Royal Society of Edinburgh, the event explores how we can make data work more efficiently in a collaborative environment.

Come along to hear how redevelopment of the OASIS form is progressing and our plans for release in Spring 2020. Redeveloped in partnership with Archaeology Scotland, the new form brings together the existing Discovery and Excavation in Scotland and OASIS forms into a single application. Amongst many new features, a new location module will greatly improve the quality of geographic information reported through the form and encourage users to upload their own boundary files straight into the map. Read more about that work here.

Find out about how the regional research frameworks are progressing in Scotland and discover how reporting your fieldwork through OASIS can help update the questions posed by the research framework panels. A short demonstration will demonstrate the use of the Research Frameworks module within OASIS (see below for a screen grab of current development version), which will allow users to pick from a geographically relevant list of questions their project informs. In turn, this allows the relevant Research frameworks panel to see which new projects are helping contribute to informing their questions.

The afternoon will explore the opportunities offered by digital data, challenging archaeologists to so much more with the digital data they create beyond a site plan fossilised in a pdf project report. Following an update from HES archives, further presentations will explore novel uses of data including how machine learning is transforming how we work. The House Age Project transforms scanned copies of historic OS maps into reusable vector maps. Read more about the project here. The afternoon ends with an update on working with Airborne Laser Scanning data and the potential of automated feature recognition.

Please register by Monday 21st October  through Eventbrite.

Putting OASIS on the map

Tim Evans + Kirsty Ackland

Everyone loves a map…

A quick look at the current development version of OASIS

One of the tasks we’re currently tackling in the OASIS redevelopment is the Location module. This is probably the most important facet of new OASIS; not only is accurate recording of a place a fundamental to everyone’s work, but within workflows project locale subsequently helps generate a lot of data in other modules of the form.

Jo has been hard at work knitting together the OpenLayers (v5) interface within a Java framework, and getting it to connect with the back-end (the classic QGIS – PostGIS stack). Without wanting to jinx things, the results are looking really good. A user can record multiple ‘sites’ within a single record, and each site can be represented by points and/or polygons. A user can also upload their own boundary files straight into the map! Once saved, a spatial query is run against an array of background mapping to generate admin areas, HER areas and (currently at review) collecting areas for Museums and Archives. Basically this tells a user where they are, and who the curators are for the record. It also allows a curator to log on and see records for ‘their’ area.

A little task we’re running in the background is coming up with ways to deal with legacy data migrated from the current database. One facet of this has been Polygons. The current OASIS database (Oracle) is not a true spatial database, and thus ‘geometries’ are stored as a set of points, but not a real polygon (if that makes sense). In order to have these polygons appear in new OASIS (PostGIS) as polygons, and not just a series of points we’ve had to do a bit of cleaning post-migration. Fortunately, we’ve been luck enough to have a fantastic Masters student (Kirsty Ackland) with us over the last few weeks. Kirsty has put my rather rusty GIS skills to shame…. In her own words:

“As part of setting up the map interface for the OASIS system an existing dataset was used to test the visualisation process. The original dataset (Oracle) is now held in a PostGreSQL/PostGIS database which was imported into QGIS. However the structure of the original data from Oracle meant the data could not be visualised as a geometry. To fix this, the dataset was run through QGIS’ database manager with a simple SQL query to retrieve only the data which was stored as polygons and output it in tabulated form to remove the geometry links. This was exported into Microsoft Excel to clean the data and remove unnecessary fields. This was then imported back into QGIS as a .csv file.

Because the geometry had been removed the records no longer had spatial references associated with them, so the next step was to find the original input co-ordinates for each record in the OASIS database. This was done using an SQL query to join of several tables within the OASIS database using the OASIS_ID, SITE_ID and GRIDREF_ID assigned to each project during data entry. This produced a table holding all of the IDs necessary to re-associate the co-ordinates (also held within this table) for each site with their assigned OASIS ID using a one-many relational join (one oasis/project ID to many gridref/co-ordinate IDs). The first dataset was then joined with the ID table using the OASIS_ID to add in the easting and northing points to their relevant project information. This was then quickly queried to remove any duplicate fields created by the process and was output into QGIS with its corrected geometry (fig 1).

Figure 1 (just the points)


To create bounding boxes for the sites held within OASIS for visualisation on the map, the convex hull tool within QGIS was used to join the co-ordinate points together a create separate polygons based on their siteID (fig 2). However, when this was complete it soon became apparent that some of the co-ordinate points were entered incorrectly resulting in the weird output seen in the screenshot where there are sharp lines running north-south. The screenshot also shows that some are in the correct place though, which suggests that this workflow works well as long as the data is correct. A secondary point to note is that some of the sites only relate to one co-ordinate point, but have been stored as “polygons” during data entry. This means although they exist within the dataset as point data, when converted to polygon to show their extent they get removed because shapefiles
can only store one datatype at a time i.e. point or polygon. The effects of this can be seen in fig 3.

Figure 2: rebuilding the polygons
Figure 3: Points + polygons

This is also an issue to bear in mind when considering the point data removed from the dataset right at the start of the process, as it is very likely some of the sites should be stored as polygons rather than points, leaving the visualised data output here incomplete.”

It’s at that point that I think we’re going to stop. We now have real functioning polygons in our new database. Some of them may be incorrect, but this is just the legacy of manual data entry (with no immediate visual control) in the current OASIS form, and which does date back to 2004. Hopefully the new system will deal with this alot more efficiently! Thanks to Kirsty for her quick and efficient help on this.

OASIS at CIfA

Just a quick blog to let everyone know that Tim and colleagues from the ADS will be at the CIfA conference in Leeds later this month.

There will be new information on the OASIS redevelopment project available from the ADS/Internet Archaeology stall, including some new promotional material .

Please come and say hello, we’re more than happy to answer any questions about what’s going on with the project. Tim will have a laptop, so (wifi permitting) he’ll be able to show you the current form as it stands!

OASIS open for testing

March saw the completion of the first major phase of the redevelopment of OASIS. After alot of hard-work by the team, we have finished the following major tasks:

  • A new database is in place (PostGreSQL + PostGIS)
  • A Java application is in place (the basis of the form) that can ‘talk’ to the database
  • Workflows mapped (what happens when(
  • Mapping old OASIS terms to existing heritage thesauri
  • Full-scale test migration of data completed
  • Testing and feedback on migration/mapping
  • The basic form is in place (at last!)

The local version has been deployed to a ‘live’ server, and a group of users who have signed up are now being notified that they can log on and have a look/test what is there.

The testing phase is now running until the end of May, during which time we will be collecting and compiling comments/bugs, as well as moving on with the rest of the build.

The slides below give a very basic example of how the new form is shaping up and the sort of information recorded within.

In a nutshell, what we have completed is…

  • A user can log on (change password etc)
  • Do all the admin type tasks detailed in this post of October 2018
  • Look at a list of their records – filter by various fields
  • View/edit/create a new record
  • Fill in details about the project (using FISH event thesaurus) and the new wordlists now hosted as LOD at heritage data, for example “development type
  • Record the location, and have this generate the geographic administrative areas (county/council, district, parish etc) and then generate the list of reviewers for that record (e.g. HER)
  • Record monument and object (and period), again using relevant FISH thesauri

What we’ll be working on next is:

  • The report / other publication section (with ability to upload a file)
  • The archives section
  • Moving forward with the map (currently only in a local version)
  • Set up the triggers that give the user an option to fill in a specialist module (currently Buildings, Geophysics and Burial Spaces Survey)

If anyone is interested in testing over the Summer, particularly for Archives and Buildings, then please get in touch with herald@ads.ac.uk