The Magic raw Field

Elasticsearch automatically creates a raw field, aka not-analyzed field, to any string typed field. This is useful as you do not always find tokenized strings useful when they are used for sorting, or aggregation in Kibana. Having a raw copy of the string content makes mentioned tasks more efficient and sometimes less confusing.

However, Elasticsearch by default creates the raw field for strings only for index whose name matches the pattern logstash-*. This is because this type of index will use the default index template which specifies the creation of the raw field. There are at least two ways to create raw fields for custom index, one is to make a custom template that works with your conf file; another, shown in this blog, is through the mappings.

The following example assumes there is a type vs in the index bedmaster. We only care to create the raw field for the string field Parameter in type vs.

  • Create the index.
PUT /bedmaster
{
  settings": {
    "number_of_shards" : 2,
    "number_of_replicas" : 1
  }
}
  • Create the mapping for vs.
PUT /bedmaster/_mapping/vs
{
    "vs": {
      "properties": {
        "Parameter": {
          "type": "string",
          "fields": {
            "raw": {
              "type":  "string",
              "index": "not_analyzed"
            }
          }
        }
      }
    }
}
  • Run logstash again to insert the data.
  • Verify the mappings are correct.
GET /bedmaster/vs/_mapping

{
  "bedmaster": {
    "mappings": {
      "vs": {
        "properties": {
          "": {
            "type": "double"
          },
          "%": {
            "type": "double"
          },
          "@timestamp": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "@version": {
            "type": "string"
          },
          "Bpm": {
            "type": "double"
          },
          "BrMin": {
            "type": "double"
          },
          "Label": {
            "type": "string"
          },
          "Location": {
            "type": "string"
          },
          "Parameter": {
            "type": "string",
            "fields": {
              "raw": {
                "type": "string",
                "index": "not_analyzed"
              }
            }
          },
          "host": {
            "type": "string"
          },
          "mm": {
            "type": "double"
          },
          "mmHg": {
            "type": "double"
          },
          "path": {
            "type": "string"
          }
        }
      }
    }
  }
}
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s