Adapt Connector to use external S3 storage for storing Einstein data
The changes are not shown correctly. To see the full diff check out: https://git.dcc.sib.swiss/sphn-semantic-framework/sphn-connector/-/compare/einstein-main...sphnc-183?from_project_id=474&straight=true
Changes are going to be merged in reference branch for Einstein einstein-main
Main changes:
- All interactions from Connector to Einstein related bucket are not executed with
boto3
(therefore ideally also supporting S3 compatible storage) - In the
.env
we can define if the storage is external or not. If it is external the connection parameters must be defined - Tested with an additional Minio storage. Connector was able to write to the external storage and notify the data to Einstein. Einstein was able to pull the data from the external storage and load it into Virtuoso.