D.4 Keeping up to date with fieldwork#

D.4.1 Existing digital resources#

Historic England maintains a record of history environment recording events often referred to as the 'Excavation Index' although it contains details of all types of fieldwork not just archaeological excavations. This was begun by the former RCHME in 1978 and it now holds details of over 80,000 archaeological interventions and other historic environment recording events in England. These include interventions dating from the earliest episodes of scientific archaeology to the present day (see C.7.3). Information has been collected from a variety of sources (including HERs) and combines bibliographic recording with direct input from reports, recording forms and microfilm archives. The scope of the Index covers both invasive (excavation, evaluation, watching brief) and non-invasive (geophysical survey, desk-based assessment, field-walking and measured survey) methods of investigation. The Index provides both a record of the event that has taken place and acts as a signpost to the archaeological features identified and the associated documentary and archive sources. Since 1998 a sub-set of the Index has been made available on the internet (Figure 27) via the ADS Catalogue (see http://archaeologydataservice.ac.uk/archives/).

Figure 31: ADS web page – Excavation Index.
Figure 31: ADS web page – Excavation Index (English Heritage and the Archaeology Data Service 2007).

From 1990 to 2012, English Heritage sponsored Bournemouth University to conduct an annual data-collection exercise, the Archaeological Investigations Project (AIP). The results of the exercise have been published as The Gazetteer of Archaeological Investigations in England, issued as an annual supplement to the British and Irish Archaeological Bibliography. The database compiled to produce these printed volumes was been deposited English Heritage.

The merger of the RCHME and English Heritage in April 1999 brought with it an opportunity to integrate the Excavation Index and The Gazetteer of Archaeological Investigations into a single resource. This concordance was made possible through the OASIS project, an alliance between English Heritage (now Historic England), the AIP and the ADS, with funding from the Research Support Libraries Program (RSLP). OASIS aims to create a single index to the grey literature of archaeological assessment reports and excavation archives in England, which will be delivered via the ADS ArchSearch catalogue The OASIS project has developed an on-line recording form and aims to promote its use by contractors so that they can submit index-level records of archaeological investigations to Historic England and the relevant HER. (See section C 7.3)

Useful website: OASIS project http://oasis.ac.uk/

The OASIS project is also in place in Scotland where it has been adopted across the sector for the reporting of archaeological projects. The OASIS project complements and encourages the continued reporting of fieldwork to the wider public through Archaeology Scotland's annual publication Discovery and Excavation in Scotland. Some HERs transfer Discovery and Excavation data into an Events module, but data is also transferred into the HES NMRS database and supplied to the relevant HER as part of an on-going data exchange programme.

In Wales the HERs have the responsibility for recording archaeological events, but the HEROS database system allows OASIS to receive event information and grey literature automatically from HER records.

D.4.2 Planning guidance and new fieldwork#

Since the advent of planning policy guidance for archaeology (PPG16 England, Circular 60/96 Wales, and NPPG 5 and PAN 42 in Scotland), archaeological consultants and field units have carried out much of the archaeological fieldwork in Britain on the basis of competitive tendering. The contractors operate to briefs or specifications developed by archaeologists working alongside the HER in local government planning offices. As well as specifying the archaeological research, briefs will normally include recommendations covering the deposit of material resulting from the research. In England this includes data and reports to be deposited in the HER, finds and archives deposited in local museums or other repositories and computer-based information to be deposited in a recognised digital archive. In Scotland the HERs require the deposition of the original site archive (for example the paper, photographic and digital records) for long-term preservation with Historic Environment Scotland, whilst copies of all reports are deposited in the local HER. For artefacts the Treasure Trove Advisory Panel (who provide independent advice to the Queen's and Lord Treasurer's Remembrancer) decides the appropriate depository.

HER officers should work with planning archaeologists to make sure that their requirements are taken into account in specifying how, when and in what format information will be deposited with the HER.

Formats for data collection#

Creating an index record to an event or archive should not place a new burden on units - it is work that many already do and that since 1997 has been supported by the Standing Conference of Archaeological Unit Managers (SCAUM). Standard record forms for collecting information from units have been developed by the Greater London HER, SCAUM, English Heritage/Historic England, the AIP, Northamptonshire HER and others. Where used, these paper forms have met with some success in improving the flow of information from data collector to HER.

However, using paper forms means that HER officers often have to re-key the information provided by the units. To minimise duplication and decrease data-entry backlogs, the use of digital forms has been explored. The OASIS form is one example, another was recommended in the ADS Guide to Good Practice for Excavation and Fieldwork Archives (http://guides.archaeologydataservice.ac.uk/). Digital versions of the paper forms described above have also been produced either as tables in word-processed documents, spreadsheets or simple databases.

The use of digital forms is not trouble free and several factors should be taken into account in agreeing specified formats for digital data in briefs. These include:

  • agreeing guidelines on how the standard recording form is to be completed
  • agreeing guidelines about the collection and provision of metadata
  • the format in which digital data is to be supplied, for example, ASCII delimited text, Access.
  • the electronic media on which data is to be transferred (see also B.10.7).

The ADS can offer advice on digital data transfers (help@ads.ahds.ac.uk). Historic England offer guidance on the use of the OASIS form (oasis@HistoricEngland.org.uk).

HER officers should set aside time to validate and enhance data. This may include:

  • confirming that the data is being supplied in accordance with core HER data standards
  • checking that the data supplied has not been corrupted during transfer and is readable
  • checking for spelling or data-input errors
  • checking the data supplied against the written reports
  • adding additional indexing information and other details.

While such forms are being introduced HER officers may need to allow time to work with both planning archaeologists and contractors to make sure that information is flowing efficiently from data collector to the HER.

D.4.3 Importing digital data from external sources#

Strategies for physically moving data between geographically removed locations#

Various forms of electronic storage media are used to move data around. An important consideration is that both sender and receiver need to use a shared technology for successful data transfer:
  • Tape has now been largely superseded as a transfer medium, mainly because few systems contain Digital Audio Tape (DAT) machines today.
  • Standard 3.5 inch floppy disks can be used to transfer small amounts of data (1.44mb).
  • Compression software can be used to produce zip files (for PCs) and tar files (for Unix systems) which increase the capacity of floppy discs. See http://www.pkware.com/ for information about PKZIP.
  • CD ROMs and DVDs can transfer up to 650mb of data for CDs and 4.7 Gb for DVDs and are becoming commonplace as CD and DVD writers become more widespread. See http://cd-info.com for information about CDs and DVDs.
  • High-capacity floppy and Zip drives may also be used. For information see http://www.winzip.com/winzip.html.
  • Files may be transferred between users as attachments to email messages. File transfer is not without its problems as many mail servers refuse files over a certain size and 'bounce' messages back to the sender. There are also security implications as e-mall attachments are notoriously vulnerable to viruses.
  • File Transfer Protocol (FTP) is an efficient way of transferring data but requires technical support. FTP is a communications program used to link source and destination systems, in which either the sender or receiver logs on to a remote system to effect a transfer. Security issues mean that many system administrators restrict access from outside although they may set up an area where files can be placed for transfer. For information about FTP see http://www.ftpplanet.com/ftpresources/basics.htm

Strategies for importing data into existing databases#

Where the data is to be imported into an existing dataset there are a number of concordance issues:
  • The new data need to be in a format acceptable for import to the software maintaining the existing database. ASCII delimited text is a standard export format for transfer into databases. In this format, a delimiting character separates the contents of the fields in a database record, for example a comma, percent or hash. Care must be taken to select a delimiter that is not present in the data. Most database applications support a number of import and export formats to facilitate the transfer of data between packages: for example as well as delimited text, Microsoft Access also supports import of data from other Access databases, Excel, Lotus, Paradox, Foxpro, dbase, Btrieve and SQL.
  • There needs to be a clear mapping between the new data and the fields that define the structure of the existing database. Where possible, the HER should specify the fields for data capture.
  • The new data should conform to any vocabulary control or data standards that have been agreed for the main database. Vocabulary lists should be supplied with the new data, particularly where codes are used. Candidate terms identified in the new data may need to be put forward for addition to existing wordlists or thesauri.
  • The new data needs to be validated and may need to be edited or manipulated to conform with agreed core-data standards for the HER using search and replace functions.

Much of this can be achieved through the use of the FISH Interoperability Toolkit (see Section B.7 for details). This comprises a suite of tools which can assist with the process of moving large amounts of data between a wide range historic environment information resources (HEIRs).

These tools will help with:

  • The provision of digital data from one organisation to another,
  • The provision of data to the public,
  • The archiving of digital data,
  • The migration of data to new systems,
  • Conformance to national standards.

Useful Websites:
FISH: http://fishforum.weebly.com/

FISH Interoperabilty Toolkit: http://archaeologydataservice.ac.uk/fishtoolkit/