Best Practices for Pushing

This article covers the "Dos and Don’ts" of interacting with Bushel’s push API. By following these guidelines, your integration with Bushel will complete with fewer surprises.

Request Tracking

Utilizing a Request ID is immensely helpful when communicating about and troubleshooting any issues with Bushel. If not provided in the initial request, Bushel will generate a random Request ID. The is recorded in Bushel’s logs and observability tools and is returned as an X-Request-ID header in the response.

Do

  • Provide a universally unique identifier in the X-Request-ID header in all requests. Record the value of this header alongside any logs or other observability records made during the request lifecycle.

  • Alternatively, record the X-Request-ID header provided in the response from Bushel.

Detecting Changes

Do

  • When data is changed, immediately trigger a push to Bushel containing only items with updated data

  • Routinely poll for changed data and only send updated data to Bushel. Polling for updates no less than once every 15m is highly recommended, particularly for frequently-updated items such as Splits and Tickets.

Don’t

Repeatedly push the entire dataset to Bushel.

Formatting Requests

Do

{
  "data": [
   ...
   "comment": "A comment with a backslash \\ in it",
   "description": "A \"quoted description\""
   ...
  ]
}

  • For a given request, group all items of the same feature together. For example:

{
  "data":[
    {
      "update-splits":{
        "splits":[
          {
            "id": 1
            ...
          },
          {
            "id": 2
            ...
          }
        ]
      }
    }
  ]
}

Don’t

  • Separate each item into its own feature, for example:

{
  "data": [
    {
      "update-splits": {
        "splits": [
          {
            "id": 1
            ...
          }
        ]
      }
    },
    {
      "update-splits": {
        "splits": [
          {
            "id": 2
            ...
          }
        ]
      }
    }
  ]
}

Detecting Deletes

Do

  • Trigger a delete message to Bushel for the item right when the item has been deleted

  • Identify deleted items during routine polling of the dataset and push a message containing deleted item(s)

  • Scrubbing

Don’t

  • Delete and reload all records from a dataset

Sending Deletes

Do

  • Use the delete operations available for all record types (e.g. delete-contracts and delete-pricing-lines)

  • If the state of the record is known, populate the deleted_at field on the object

Don’t

  • Generate a record with the deleted_at field set with fake data for all other required fields.

Timestamps

Do

  • All timestamps are sent in RFC 3339 format with the appropriate timezone offset

  • Setting unknown timestamps to null (where acceptable) or a static time (e.g. set created_at to 1970-01-01T00:00:00Z)

Don’t

  • Send timestamps in local time with an incorrect offset. For example, if local time is 8:00a in Chicago do not format the timestamp as 2021-10-08T08:00:00+00:00 (i.e. 8:00a UTC). This will result in incorrect timestamps being displayed to the user in the app.

  • Set updated_at to the current time

Identifiers

In order for relationships between records to function, the identifiers used must match exactly. For example, the id of an account must be identical to the corresponding user_id on a Contract in order for a User to view their Contract.

Do

  • Ensure all identifiers (e.g. id, user_id, ticket_id) have consistent casing across all record types

  • Ensure all identifiers have trimmed (or at a minimum, consistent) trailing whitespace

Historical Data

Syncing of all records that have been updated since a particular start date is required for the product to perform correctly.

Do

  • Send all data for all producers that has been updated since a configurable date. Bushel typically recommends sending at least 2 growing seasons of data, however this is dependent upon your use cases and customer preferences

Re-pushing Data

Having the ability to re-push historical data is vital in case bugs or errors are identified in the integration. Additionally, under exceedingly rare circumstances, Bushel may request data of a certain type to be re-pushed.

Do

  • Have the ability to re-push all data for each record type

Error Handling

We take pride in our uptime and strive to always have our services available. Unfortunately, errors both within and outside our control do occur. Building an integration resilient to intermittent problems reduces support and manual recovery efforts while providing a better user experience to our customers.

Do

  • If a retry-able error is received when making a request, the request should be retried after a delay calculated by an exponential backoff algorithm. Retry-able errors include:

    • 409 errors (which may be returned by our firewall)

    • All 500-level errors

Don’t

  • Immediately retry failed requests