Bulk adding tags to files via REST API without a policy

We use a SQL select query to fetch files from the Spectrum Discover metaocean table and let a script work on these files. The script produces a JSON file with the fkey of every file and a JSON dictionary with tags that we want to add. Since we're not using the policyengine of Spectrum Discover we are looking for a performant way to upload these JSON documents into the DB2 database.

Our proposal is to have a REST API extension for Spectrum Discover that accepts JSON documents in the following format:

{

"fkey": "uk_vf085_v01uk_vf085_v01:/vol/uk_vf085_v01/test-nfs-share7699699",

"tags": {

"tag1": "This is the value for tag1",

"tag2": "This is the value for tag2",

"tag3": "This is the value for tag3"

}

},

{

"fkey": "uk_vf085_v01uk_vf085_v01:/vol/uk_vf085_v01/test-nfs-share11057739",

"tags": {

"tag1": "Another value for tag2",

"tag10": "value for tag10",

"MD5": "112d8f46312998a9d8b67aedd2365522"

}

}

The workflow is:

• Initial scan finds files on a share and puts them into the metaocean table

• We run a SQL query to get all these files using the already existing fkey as the unique id for every file

• In the next step instead of using the Python SDK we run whatever script or binary is the best and most performant tool to extract data from files and produce the above mentioned JSON document

• The new REST API endpoint takes the JSON document and insert its entries into DB2