Demyst services
Understand the services Demyst delivers to help you maximise the value of the Demyst Platform
Services to support the Demyst Platform
The Demyst Platform is supported by a variety of services, supporting your team in managing all parts of the external data lifecycle. This includes use case ideation, evaluation and testing, data due diligence, production deployment and ongoing maintenance, and more, all with the aim to maximise the value you get from the Demyst Platform. An indicative set of those services is diagramed below, with details and benefits followed in the table after, split out by capabilities. Included services may vary in your licence with Demyst.
Indicative list of services to complement the Demyst Platform
Service components
Broad groupings of services capabilities.
| Service component | Definition | Capability |
|---|---|---|
| Setup: Architecture and tooling alignment | Agree on core architectural and tool standards: Identify approved ingress and egress patterns, document the tools used in your centralised data team's stack, and align on the required infrastructure, security, and integration aspects. | - Identification of approved ingress and egress patterns - Documentation of tools - Evaluating integration and interoperability - Establishing monitoring and alerting mechanisms |
| Setup: Process alignment | Agree on ways of working standards: Understand and optimise various processes related to third-party data acquisition and management, including data access controls, the role of data stewards, onboarding new datasets, managing data provider relationships, and handling data requests from consumers. | - Documenting the current ingestion process - Agreeing on data cataloging and metadata management system - Defining data access policies and permissions - Designing a data provider onboarding workflow - Agreeing on data provider management best practices |
| Setup: Hub integration | Establish an integration between Demyst & client Data Hubs: Integrate the selected tools and technologies, confirm end-to-end connectivity, and obtain stakeholder approval before moving to the operationalisation phase. | - Integrating tooling - Connectivity test - Sign off and operationalisation |
| Art of the possible | Provide an avenue for SME knowledge transfer about potential industry use cases: A Data Hub needs to build out a backlog of use cases solvable with external data that deliver value across all the spokes of an enterprise. Demyst supports through recommendation of use cases that we see across the industry, and can host workshops with the client spoke team to tease out additional use cases. | - Use case recommendations |
| Data evaluation | Remove blockers that commonly make data evaluation a bottleneck: Demyst delivers a variety of services for evaluation in order to expedite the time between data identification and access to evaluation data for a client. We do this through procuring and diligence evaluation data on your behalf with data providers, then subsequently delivering it in a standardised manner. | - Evaluation procurement - Evaluation due diligence - Evaluation data ingestion & delivery - Evaluation data onboarding report |
| Production delivery | Leverage the Demyst Platform to deliver a robust tailored production data pipeline: Demyst delivers a variety of services for production data pipe creation, with our services team leveraging the Demyst Platform to provide the right metadata and deployment configuration to suit the customer’s needs, and taking on the pain of engaging with upstream data providers. | - Production procurement accelerator - Data provider technical requirement gathering - Connector configuration - Connector metadata management - Data API & Data Share deployment configuration - Matching support for data enrichment |
| Maintenance | Demyst takes on the support burden, and delivers best practice maintenance: Let Demyst handle the pains of upstream support with all your data providers, simplified through our monitoring and alerting functionality. Rest assured that you’re receiving the most up to date versions of available data and metadata. | - Proactive monitoring, alerting and SLA management - Incident remediation - Connector and deployment upgrades and management |
Service capabilities
Details of all standard services capabilities and the value they offer.
Jump to setup specific services
Jump to core services
Setup specific services
| Capability | Description | Benefit |
|---|---|---|
| Identification of approved ingress and egress patterns | This process ensures that data is transferred in and out of the Data Hub securely and efficiently with patterns approved by the enterprise. | - Required to setup Demyst |
| Documentation of tools | Document tools such as the catalog, data warehouse or data lake utilised by the centralised data team. It's essential to maintain a comprehensive record of the tools and technologies used in the Data Hub stack. This documentation should describe how each tool contributes to the overall workflow and enhances the value of the data management processes. | - Required to setup Demyst |
| Evaluating integration and interoperability | Analyse existing systems and applications to determine the best methods for integrating them with the Data Hub, ensuring seamless communication and data exchange between all components. | - Required to setup Demyst |
| Establishing monitoring and alerting mechanisms | Identify the key performance indicators to track and create monitoring systems to ensure the Data Hub's health and performance are consistently maintained. | - Required to setup Demyst |
| Documenting the current ingestion process | Create a detailed record of the existing methods and procedures for ingesting external data into the system, which can serve as a baseline for designing improvements and enhancements. | - Required to setup Demyst |
| Agreeing on data cataloging and metadata management system | Implement a system for populating the enterprise data catalog available, making it easier for users to discover and access relevant data. | - Required to setup Demyst |
| Defining data access policies and permissions | Establish clear guidelines for granting data access to different user roles within the organisation, ensuring that sensitive data is appropriately protected and accessed only by authorised personnel. | - Required to setup Demyst |
| Designing a data provider onboarding workflow | Develop a process for setting up new data provider relationships, including steps for legal contract negotiations, data ingestion setup, and integration with the Data Hub. | - Required to setup Demyst |
| Agreeing on data provider management best practices | Create guidelines and procedures for maintaining ongoing relationships with data providers, addressing aspects such as communication, performance monitoring, and issue resolution. | - Required to setup Demyst |
| Integrating tooling | Implement the connections and configurations required to integrate the selected tools and technologies, as determined during the Architecture and Tooling Alignment phase. | - Required to setup Demyst |
| Connectivity test | Use sample data that adheres to the agreed-upon interfaces to test the end-to-end connectivity and functionality of the integrated tools. | - Required to setup Demyst |
| Sign off and operationalisation | Once the integration has been thoroughly tested and validated, obtain approval from relevant stakeholders to confirm that the Data Hub meets the organisation's requirements and expectations. | - Required to setup Demyst |
Core services
| Capability | Description | Benefit |
|---|---|---|
| Use case recommendations | Demyst supports the Data Hub team with frameworks and workshops to build out an extensive roadmap of data sources to onboard, covering both net new data and refactoring of existing. Demyst can provide recommendations about successful industry wide use cases that could drive value to a client, providing inspiration for initiatives that your business teams can action to drive value & competitive advantages. Demyst supports this SME knowledge with collected metadata about specific Connector performance against various initiatives. | - Leverage Demyst's expansive external data knowledge to learn what is being used across your entire industry - Build a deeper backlog despite Data Hub team capacity constraints |
| Evaluation procurement | Demyst negotiates (typically free) POC terms and enters NDA/POC agreement directly with the data provider to deliver evaluation data to client without them needing such direct agreements. | - Demyst takes on this activity for your team reducing the impact of capacity constraints on evaluation timelines or the number of sources that can be tested - Avoid the headache of working with data providers that are slow and reticent to respond to diligence requests and/or require follow-up to properly complete the process - Demyst does this to mutually agreed standards specifically focused on evaluation in order to ensure that everything is conducted appropriately without any unnecessary risk introduced such as data misuse or access to data that was collected inappropriately or unethically. Determine any concerns with this prior to conducting time-intensive analytics or production setup steps - By providing this standardised and streamlined process, Demyst helps ensure that there is a path that internal stakeholders can leverage to test data in a reasonable timeframe without feeling the need to go direct to the data provider or engage in any "cowboy" methods (e.g. being emailed the data from the provider) |
| Evaluation due diligence | Demyst conducts a standardised in-depth due diligence process (privacy, security, compliance, data ethics assessment) for given use case and data source(s). | See "evaluation procurement" benefits |
| Evaluation data ingestion & delivery | Demyst creates a metadata rich Connector populated with evaluation data from the data provider or access to an upstream API enabled for evaluation. Demyst configures an evaluation deployment (Data API or Data Share) in the standardised agreed manner with the client for evaluation purposes, delivering the data to them for analytics so client can make a go/no-go decision on the data. This process may involve data enrichment matching exercise conducted by Demyst. | See "evaluation procurement" benefits |
| Evaluation data onboarding report | Demyst to generate and provide a report to summarise key information about the data being evaluated. Includes key metadata and information gathered from the procurement (such as approximate production commercials), diligence (such as outputs of reviews related to ethics, information security, regulatory compliance, privacy, permitted use and more) and data ingestion processes (such as key data attributes and quality statistics). Provided by Demyst delivery team to the client so they can determine if there are any red flags or poor quality indicators that would prohibit data use prior to spending time and effort analysing the data for uplift. | - Identify any red flags related to the data itself, commercial terms or diligence details (such as how it was collected) prior to spending time analysing it, and before moving to a production process. This de-risks each item in the backlog of data being considered for entering the Data Hub by moving these checks forward, avoiding wasted effort on something that is later identified as not viable. This means your production backlog of data can be more focused on a data with a higher likelihood of success - Simplify decision making on evaluation data before analytics has occurred in a single report |
| Production procurement accelerator | This additional optional service is where Demyst will diligence and procure production data on behalf of a given client. This means: 1) We procure and diligence the data to a mutually agreed level of satisfaction on behalf of the client 2) We will then commercially act as a reseller of that data to the client (associated data fees are applicable) 3) We will provide the results of our procurement and diligence process to the client to support their transition to a direct relationship (with Demyst as processor) This service is focused on expediting a client's time to production data and so the scenarios where this makes sense for a client is: 1) Time bound product launches that require a given external data source in production 2) Time-bound analytics for a one-off brick of data This service is typically delivered for an additional yearly fee, with the intention that the client's transition to a direct processor relationship within a year. The transition is usually commercially sensible given the client's stronger bargaining power on data fees compared to Demyst. | Demyst enables your organisation to rapidly expand its external data portfolio to meet timelines of particular use cases through a standardised procurement path through Demyst. This means you can experience: 1) Faster time to market for net new data sources in production 2) No additional margin on data you’re consuming from Demyst over the long term (assuming you transition to being processor directly in the medium term) |
| Data provider technical requirement gathering | Demyst will gather the API or file based ingestion details required for connector creation by engaging with the data provider. This is a long complex list of technical and non-technical requirements such as SFTP address, authentication, certificates, dictionaries, retry mechanisms, encoding, encryption, record validation and more for file based connectors, and similarly API specs, authorisation, error handling test cases, credentials and certificate management, rate limiting, and more for API based connectors. | - Avoid the headache of working with data providers that are slow and reticent to respond. This commonly takes 2-8 weeks due to lots of back and forth with the data provider - Demyst speeds up this process through a standardised engagement method to ensure no information is missed and to reduce back and forth |
| Connector configuration | Demyst will leverage the Demyst Platform to create a Connector for you that integrates with an upstream data source based off the gathered technical and non-technical requirements from the data provider. | - Demyst services team, leveraging Demyst Platform is able to create a Connector, rich with metadata and enabling access to an upstream data source that then fuels a customer deployment, either Data API or Data Share in an agreed, easy to maintain manner. Connector configuration is a core foundation of the Demyst Platform |
| Connector metadata management | Demyst Connector are metadata rich and with metadata simultaneously gathered directly from data providers and automatically generated from Connector creation activities. This is a service ran by Demyst's team to populate a new Connector and maintain that over time, with all the relevant metadata that then feeds upstream client catalog systems. | - Demyst takes on this activity for your team reducing the impact of capacity constraints on metadata completeness or recency, enabling your team to focus on other higher value work - Avoid the headache of working with data providers that are slow and reticent to respond when gathering metadata for your catalog |
| Data API & Data Share deployment configuration | Leverage the Demyst Platform to create a configured deployment for data accessed through Data API or Data Share. For Data API, this includes creating solution logic for data provider orchestration such as combining data into a single API or applying logic like water-falling or failover, and output schema standardisation, and API management activities like secret/keys, caching, alerting mechanisms, documentation, sandbox API, versioning, and more. For Data Share, this includes setting configuration to meet agreed in advance standards of destination landing zone (e.g. schema, file format, encryption, encoding, etc.), and management activities like scheduling, retry mechanisms, creating data quality checks, schema evolution detection and management, and more. | - Demyst services team, leveraging Demyst Platform is able to configure a tailored deployment for a customer, combining Connectors, appropriate logic, and ETL to standardise to the customer's landing zone needs, conducted in a manner that supports scalable and secure maintenance. Deployment configuration is a core foundation of the Demyst Platform - As part of this process, Demyst can apply workflow optimisations including how data sources are optimally combined to achieve maximum impact, meet SLAs, and optimise costs |
| Matching support for data enrichment | Leveraging Demyst's entity resolution technology, Demyst can enrich a client provided input file with additional data from Connectors, with the enriched file (the original inputs plus additional new rows) returned back to them through a Data Share. This involves QA on the output file to ensure maximised match rates. | - Maximise match rates between your internal data and the external data ecosystem |
| Proactive monitoring, alerting and SLA management | Demyst's Global Service Desk monitors key performance metrics such as uptime/availability, latency, error rates, and more for all deployed Connectors, with alerting setup for exceeded thresholds. This data is also delivered to clients via Demyst administration APIs into their appropriate logging systems, but this service is specifically for Demyst's support functions, alerting for the need of any upstream remediation efforts with data providers. This also enables Demyst to identify areas of solution performance optimisation for the client, as well as validate the performance of data providers against known SLAs, flagging any concerns. | - Timely detection and resolution of issues impacting use cases reduces downtime of solutions - Demyst is an expert in managing the signal to noise ratio for Data API error monitoring - Demyst centralises all the tracking and communication of planned data provider downtime rather than dealing with the complexity of disparate information from multiple providers |
| Incident remediation | Leveraging monitoring and alerting systems, Demyst's Global Service Desk inspects any deployments experiencing errors or low performance, identifies causes and then orchestrates remediation efforts with all impacted upstream data providers and downstream clients. This is made possible through Demyst's complete visibility of support and SLA structures across all data providers, and the monitoring and alerting systems established for each deployed solution when they are configured through the Demyst Platform. | - Demyst provides effective and timely issue triage, identifying what is broken quickly and obfuscating away from you the complexity of managing support from multiple upstream providers with different SLAs and support structures |
| Connector and deployment upgrades and management | Demyst upgrades it's Connectors on a regular proactive basis as new versions of APIs and bulk data are available. This is managed through the Demyst Platform with a version control system separately for Connectors and downstream deployments so that a customer can be transitioned to new versions of a Connector seamlessly without a downstream integration or schema change. If a change is necessary and desired by the client (such as a new data attribute), then this change is actioned when mutually agreed to a new version of that deployment, with the old version managed in parallel during that transition, providing confidence in deployment uptime. | - Avoid changes that break downstream systems - Reduce maintenance efforts |
Updated 9 months ago