Contact us
Go to platform

Top-level functions

Applying complex operations to the available data in your Data API

Suggest Edits

$firstOf

  • Description: Returns the first non-empty attribute of list of attributes.
  • Arguments: A list of paths to connectors + attributes.
  • Returns: An attribute, which will be serialized in JSON.
{
  "$firstOf":
  	[
      "equifax_austin_tetra_details.primary_sic.sic_code",   
      "hosted_infogroup_business_places.results[0].primary_sic_code_id"
    ]
}

$allEmpty

  • Description: Returns true if all of the given attributes are empty.
  • Arguments: A list of paths to connectors + attributes.
  • Returns: true/false.
{
  "$allEmpty": 
  	[ 
    	"equifax_austin_tetra_details.primary_sic.sic_code", 
      "hosted_infogroup_business_places.results[0].primary_sic_code_id"
    ]
}

$anyEmpty

  • Description: Returns true if any of the given attributes are empty.
  • Arguments: A list of paths to connectors + attributes.
  • Returns: true/false.
{
  "$anyEmpty": 
  	[ 
    	"equifax_austin_tetra_details.primary_sic.sic_code", 
      "hosted_infogroup_business_places.results[0].primary_sic_code_id"
    ]
}

$map

  • Description: Replace upstream attribute values with custom ones. Usable in combination with other top-level functions.
  • Arguments: A mapping from upstream → custom attributes.
  • Returns: The custom attribute, or the upstream attribute if no argument matched.

📘

$map needs to be called together with another function. If you only want to change a single attribute, you can use with the $at function.

# Mapping with the output of $firstOf
{
  "$firstOf": [
    "domain_from_email.details.demographics.living_status",
    "provider_5.home_owner_status"
  ],
  "$map": {
    "r":         "renter",
    "rent":      "renter",
    "o":         "owner",
    "homeowner": "owner",
    "own":       "owner"
  }
}

# This update allows us to map the output of the individual attributes that are passed into 
# $firstOf. The example below shows that for the same code, C006 from Quantarium and Attom, 
# different values could be mapped.         
{
  ...,
  "refine": {
    "route_demo": {
      "$firstOf": [
        {
          "$at": "hosted_attom_residential_tax_assessor.results[0].contact_owner_mail_address_crrt",
          "$map": {
            "C006": "City"
          }
        },
        {
          "$at": "hosted_quantarium_open_lien.results[0].pa_carrier_route",
          "$map": {
            "C006": "County"
          }
        }
      ],
      "$map": {
        "C001": "Street",
        "C002": "Road"
      }
    }
  }
}

$at

  • Description: Returns the attribute at the given path.
  • Arguments: An attribute path in full, i.e. connector + attribute.
  • Returns: The attribute. Note: This function is particularly useful used together with the $map function.
# This example uses a connector which has a construction type attributes that has many possible
# values. These values and their descriptions are within the Data Dictionary. Instead of 
# frequently referring back to the Data Dictionary, we can use $at with $map for easier access 
# to all or a subset of this enumeration.

{
  ...,
  "refine": {
    "construction_types": {
      "$at": "hosted_attom_commercial_tax_assessor.results[*].construction",
      "$map": {
         "0": "Unknown construction type",
         "1": "Type isn't specified",
         "10": "Wooden construction",
         "31": "Brick construction"
      }
    }
  }
}


# The refined output informs you immediately of what construction types
# were returned at the attribute, for all matched results.
{
  "refine": {
    "construction_types": "Wooden construction"
  }
}

$extract

  • Description: Extracts attributes from a list, while allowing the user to rename them and optionally apply a function to them.
  • Arguments:
    • Required
      • list_path: A path to a list. May not contain wildcards.
      • select: A JSON object (map, dictionary, ...) containing either.
        • The $at function, which takes an attribute path: Nested attributes may be retrieved with dot notation, i.e. address.post_code.
        • The $const function, which takes a scalar JSON node (string, number, boolean, or null) to which it will be applied to each list item.
    • Optional
      • drop_nulls: A Boolean indicating that keys whose values are null (empty) be removed from the dictionary.
      • flatten: A key in the dictionary to be selected and returned within a list, without wrapping as a dictionary.
  • Returns: The custom schema.
# Example - Using only $at

# Assuming the following connector output:
{
  "companies_house_company_search": {
    "data": {
      "items": [
        {  "company_status": "dissolved", "company_type": "private-unlimited-nsc" },
        {  "company_status": "closed", "company_type": "oversea-company" }
      ]
    }
  }
}


# The $extract function is called like so, in refine:
{
  ...,
  "refine": {
    "new_schema": {
      "$extract": {
        "list_path": "companies_house_company_search.items",
        "select": {
          "status": { "$at": "company_status" },
          "type":   { "$at": "company_type" }
        }
      }
    }
  }
}

# To yield the below final result:
{
  "refine": {
    "new_schema": [
      { "status": "dissolved", "type": "private-unlimited-nsc" },
      { "status": "closed", "type": "oversea-company" }
    ]
  }
}

# Example - Using $at and $const

# Assuming a similar connector output as in example 1,
# we could call $extract in the refine section, like this:
{
  ...,
  "refine": {
    "new_schema": {
      "$extract": {
        "list_path": "companies_house_company_search.items",
        "select": {
          "new_attribute_name": { "$const": "Companies House - Demyst" },
          "number": { "$at": "company_number" },
          "status": { "$at": "company_status" }
        }
      }
    }
  }
}

# The result will look as follows:
{
  "refine": {
    "new_schema": [
      { "new_attribute_name": "Companies House - Demyst",
        "number": "123456789",
        "status": "dissolved"
      },
      { "new_attribute_name": "Companies House - Demyst",
        "number": "987654321",
        "status": "closed"
      }
    ]
  }
}      

# Example - Using $at with flatten

# Still assuming a similar connector output as in example 1,
# we could call $extract in the refine section, like this:
{
  ...,
  "refine": {
    "my_b2b_schema": {
      "$extract": {
        "list_path": "companies_house_company_search.items",
        "select": { "send_business_mail_to": { "$at": "address_snippet" } },
        "flatten": "send_business_mail_to" },
    "my_kyb_schema": {
      "$extract": {
        "list_path": "companies_house_company_search.items",
        "select": {
          "new_attribute_name": { "$at": "title" },
          "number": { "$at": "company_number" },
          "status": { "$at": "company_status" }
        }
      }
    }
  }
}

# The response would then look something like the following
{
  "refine": {
    "my_b2b_schema": [
      "123 First Line Address Rd, New Yorkie, NY, USA",
      "ABC 2nd Street #ZYX, Big State, TX, USA"
    ],
    "my_kyb_schema": [
      { "new_attribute_name": "DEMYST LIMITED",
        "number": "123456789",
        "status": "active"
      },
      { "new_attribute_name": "DEMYST CONSULTING LIMITED",
        "number": "987654321",
        "status": "active"
      }
    ]
  }
}

$dictionary

  • Description: $dictionary introduces a new level of nesting inside of $refine, which we currently can’t support natively without using $dictionary due to syntactic ambiguity.
  • Arguments:
    • Supported right-hand side arguments are all the top-level functions as defined in this guide.
    • After the first call of $dictionary, make sure to use the keyword dictionary (without $) after the colon, for type casting.
  • Returns: A new dictionary.
# Assuming the following connector output:
{
  "companies_house_company_search": {
    "data": {
      "items": [
        {  "company_status": "dissolved", "company_type": "private-unlimited-nsc" },
        {  "company_status": "closed", "company_type": "oversea-company" }
      ]
    }
  }
}

# We can create a complex output structure by combining multiple top level functions along with 
# the $dictionary function as such
{
  ...,
  "refine": {
    "my_special_dict:dictionary": {
      "$dictionary": {
        "company_likely_name": {
          "$dictionary": {
            "take_first_title": { "$at": "companies_house_company_search.items[0].title" }
          }
        },
        "my_sub_dict": {
          "$dictionary": {
            "my_sub_2_dict": {
              "$dictionary": {
                "my_sub_3_dict": {
                  "$extract": {
                    "list_path": "companies_house_company_search.items",
                    "select": { "number": { "$at": "company_number" } },
                    "flatten": "number"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

# The above logic yields an output like so:
{
  "refine": {
    "my_special_dict": {
      "company_likely_name": {
        "take_first_title": "DEMYST LIMITED"
      },
      "my_sub_dict": {
        "my_sub_2_dict": {
          "my_sub_3_dict": [
            "0123456789",
            "9876543210",
            "0192837465"
          ]
        }
      }
    }
  }
}

Updated 9 months ago