Elasticsearch

With Elasticsearch, it's easy to hit the ground running. When I built my first Elasticsearch cluster, it was ready for indexing and search within a matter of minutes. And while I was pleasantly surprised at how quickly I was able to deploy it, my mind was already racing towards next steps. But then I remembered I needed to slow down (we all need that reminder sometimes!) and answer a few questions before I got ahead of myself. Questions like:

• How confident am I that this will work in production?
• What is the throughput capacity of my cluster?
• How is it performing?
• Do I have enough available resources in my cluster?
• Is it the right size?

If these sound familiar, great! These are questions all of us should be thinking about as we deploy new products into our ecosystems. In this post, we'll tackle performance Elasticsearch benchmarking and sizing questions like the above. We’ll go beyond “it depends” to equip you with a set of methods and recommendations to help you size your Elasticsearch cluster and benchmark your environment. As sizing exercises are specific to each use case, we will focus on logging and metrics in this post.

Think like a service provider

When we define the architecture of any system, we need to have a clear vision about the use case and the features that we offer, which is why it’s important to think as a service provider — where the quality of our service is the main concern. In addition, the architecture can be influenced by the constraints that we may have, like the hardware available, the global strategy of our company and many other constraints that are important to consider in our sizing exercise.

Note that with the Elasticsearch Service on Elastic Cloud, we take care of a lot of the maintenance and data tiering that we describe below. We also provide a predefined observability template along with a checkbox to enable the shipment of logs and metrics to a dedicated monitoring cluster. You can tweak the sizing to match the calculations you come up with below. Feel free to spin up a free trial cluster as you follow along.

Computing resource basics

Performance is contingent on how you're using Elasticsearch, as well as what you're running it on. Let's review some fundamentals around computing resources. For each search or indexing operation the following resources are involved:

Storage: Where data persists

• SSDs are recommended whenever possible, in particular for nodes running search and indexi operations. Due to the higher cost of SSD storage, a hot-warm architecture is recommended to reduce expenses.
• When operating on bare metal, local disk is king!
• Elasticsearch does not need redundant storage (RAID 1/5/10 is not necessary), logging and metrics use cases typically have at least one replica shard, which is the minimum to ensure fault tolerance while minimizing the number of writes.

Memory: Where data is buffered

• JVM Heap:Stores metadata about the cluster, indices, shards, segments, and fielddata. This is ideally set to 50% of available RAM.
• OS Cache:Elasticsearch will use the remainder of available memory to cache data, improving performance dramatically by avoiding disk reads during full-text search, aggregations on doc values, and sorts.

Compute: Where data is processed

Elasticsearch nodes have thread pools and thread queues that use the available compute resources. The quantity and performance of CPU cores governs the average speed and peak throughput of data operations in Elasticsearch.

Network: Where data is transferred

The network performance — both bandwidth and latency — can have an impact on the inter-node communication and inter-cluster features like cross-cluster search and cross-cluster replication.

Sizing by data volume

For metrics and logging use cases, we typically manage a huge amount of data, so it makes sense to use the data volume to initially size our Elasticsearch cluster. At the beginning of this exercise we need to ask some questions to better understand the data that needs to be managed in our cluster.

• How much raw data (GB) we will index per day?
• How many days we will retain the data?
• How many days in the hot zone?
• How many days in the warm zone?
• How many replica shards will you enforce?

In general we add 5% or 10% for margin of error and 15% to stay under the disk watermarks. We also recommend adding a node for hardware failure.

Let’s do the math

• Total Data (GB) = Raw data (GB) per day * Number of days retained * (Number of replicas + 1)
• Total Storage (GB) = Total data (GB) * (1 + 0.15 disk Watermark threshold + 0.1 Margin of error)
• Total Data Nodes = ROUNDUP(Total storage (GB) / Memory per data node / Memory:Data ratio)
  In case of large deployment it's safer to add a node for failover capacity.

Example: Sizing a small cluster

You might be pulling logs and metrics from some applications, databases, web servers, the network, and other supporting services . Let's assume this pulls in 1GB per day and you need to keep the data 9 months.

You can use 8GB memory per node for this small deployment. Let’s do the math:

• Total Data (GB) = 1GB x (9 x 30 days) x 2 = 540GB
• Total Storage (GB)= 540GB x (1+0.15+0.1) = 675GB
• Total Data Nodes = 675GB disk / 8GB RAM /30 ratio = 3 nodes

Let’s see how simple it is to build this deployment on Elastic Cloud:

Example: Sizing a large deployment

Your small deployment got successful and more partners want to use your Elasticsearch service, you may need resizing your cluster to much the new requirements.

Let’s do the math with the following inputs:

• You receive 100GB per day and we need to keep this data for 30 days in the hot zone and 12 months in the warm zone.
• We have 64GB of memory per node with 30GB allocated for heap and the remaining for OS cache.
• The typical memory:data ratio for the hot zone used is 1:30 and for the warm zone is 1:160.

If we receive 100GB per day and we have to keep this data for 30 days, this gives us:

• Total Data (GB) in the hot zone = (100GB x 30 days * 2) = 6000GB
• Total Storage (GB) in the hot zone = 6000GB x (1+0.15+0.1) = 7500GB
• Total Data Nodes in the hot zone = ROUNDUP(7500 / 64 / 30) + 1 = 5 nodes
• Total Data (GB) in the warm zone = (100GB x 365 days * 2) = 73000GB
• Total Storage (GB) in the warm zone = 73000GB x (1+0.15+0.1) = 91250GB
• Total Data Nodes in the warm zone = ROUNDUP(91250 / 64 / 160) + 1 = 10 nodes

Let’s see how simple it is to build this deployment on Elastic Cloud:

Benchmarking

Now that we have our cluster(s) sized appropriately, we need to confirm that our math holds up in real world conditions. To be more confident before moving to production, we will want to do benchmark testing to confirm the expected performance, and the targeted SLA.

For this benchmark, we will use the same tool our Elasticsearch engineers use Rally. This tool is simple to deploy and execute, and completely configurable so you can test multiple scenarios. Learn more about how we use Rally at Elastic.

To simplify the result analysis, we will split the benchmark into two sections, indexing and search requests.

Indexing benchmark

For the indexing benchmarks we are trying to answers the following questions:

• What is the maximum indexing throughput for my clusters?
• What is the data volume that I can index per day?
• Is my cluster oversized or undersized ?

For the purpose of this benchmark we will use a 3 nodes cluster with the following configuration for each nodes:

• 8 vCPU
• Standard persistent disk (HDD)
• 32GB/16 heap

Indexing benchmark #1:

The data set used for this benchmark is Metricbeat data with the following specifications:

• 1,079,600 documents
• Data volume: 1.2GB
• AVG document size: 1.17 KB

The indexing performance will depend also on the performance of the indexing layer, in our case Rally. We'll execute multiple benchmark runs to figure out the optimal bulk size and the optimal thread count in our case. 

We’ll start with 1 Rally client to find the optimal batch size. Starting from 100, then increasing by double in subsequent runs, which shows that we have an optimal batch size of 12k (around 13.7 MB) and we can reach 13K events indexed per second.

Next, using a similar method, we find that the optimal number of clients is 16 allowing us to reach 62K events indexed per second.

Our cluster can handle a maximum indexing throughput of 62K events per second. To go further we would need to add a new node.

I was curious to see how many indexing requests one node can handle, So I’ve executed the same track on a one node cluster and a two node cluster to see the difference.

Conclusion: For my test environment, the maximum indexing throughput is...

• With 1 node and 1 shard we got 22K events per second.
• With 2 nodes and 2 shards we got 43k events per second.
• With 3 nodes and 3 shards we got 62k events per second.

Any additional indexing request will be put in the queue, and when the queue is full the node will send rejection responses.

Our dataset impacts the cluster performance, which is why it’s important to execute the tracks with your own data.

Indexing benchmark #2:

For the next step, I have executed the same tracks with the HTTP server log data using the following configuration:

• Volumes: 31.1GB
• Document: 247,249,096
• Avg document size: 0.8 KB

Optimal bulk size is 16K documents.

Yielding an optimal number of 32 clients.

And the maximum indexing throughput for the http server log data is 220K events per second.

Search benchmark

Intro sentence about why we have to test search as well. To benchmark our search performance, we will consider using 20 clients with a targeted throughput of 1000 OPS.

For the search, we will execute three benchmarks:

1. Service time for queries

We will compare the service time (90 percentiles) for a set of queries.

Metricbeat data set

• auto-date-historgram
• auto-data-histogram-with-tz
• date-histogram
• Date-histogram-with-tz

We can observe that auto-data-histogram-with-tz query have the higher service time.

HTTP server log data set

• Default
• Term
• Range
• Hourly_agg
• Desc_sort_timestamp
• Asc_sort_timestamp

We can observe that desc_sort_timestamp and desc_sort_timestamp queries have the higher service time.

2. Service time for parallel queries

Let's see the 90 percentile service time increased if I execute the queries in parallel.

3. Index rate and service time with parallal indexing

We'll execute a parallel indexing task and the searches to see the indexing rate and the service time for our queries.

Let's see the 90 percentile service time for the queries increased when executing in parallel with indexing operations.

Reading our results

With 32 clients for indexing and 20 users for search we have:

• Indexing throughput 173K, less than 220K noted previously.
• Search throughput 1000 events per second.

Conclusion

The sizing exercise equips you with a set of methods to calculate the number of nodes that you need based on data volume. In order to best plan for the future performance of your cluster, you will also need to benchmark your infrastructure — something we've already taken care of in Elastic Cloud. Since the performance will depend also on your use case, we recommend using your data and your queries to be closest to the reality. More details about how to define a custom workload in the Rally documentation.

Welcome back! In the last blog we set up bidirectional communication between ServiceNow and Elasticsearch. We spent most of our time in ServiceNow, but from here on, we will be working in Elasticsearch and Kibana. By the end of this post, you'll have these two powerful applications working together to make incident management a breeze. Or at least a lot easier than you may be used to!

As with all Elasticsearch projects, we will create indices with mappings that suit our needs. For this project, we need indices that can hold the following data:

• ServiceNow incident updates: This stores all information coming from ServiceNow to Elasticsearch. This is the index that ServiceNow pushes updates to.
• Application uptime summary for ease of use: This is going to store how many total hours each application has been online. Consider this an intermediate data state for ease of use.
• Application incident summary: This is going to store how many incidents each application has had, each application’s uptime, and the current MTBF (mean time between failure) for each application.

The last two indices are helper indices so that we don’t have to have a whole bunch of complicated logic running every time we refresh the Canvas workpad we'll create in part 3. They are going to be updated continuously for us through the use of transforms.

Create the three indices

To create the indices, you'll use the below guidance. Note that if you don't use the same names as used below, you might need to make adjustments in your ServiceNow setup.

servicenow-incident-updates

Following best practices, we are going to set up an index template, index alias, and then an index lifecycle management (ILM) policy. We will also create an index template so that the same mapping is applied to any future index created by our ILM policy. Our ILM policy is going to create a new index once 50GB of data is stored within the index and then delete it after 1 year. An index alias is going to be used so we can easily point towards the new index when it’s created without updating our ServiceNow business rule. 

# Create the ILM policy 
PUT _ilm/policy/servicenow-incident-updates-policy 
{ 
  "policy": { 
    "phases": { 
      "hot": {                       
        "actions": { 
          "rollover": { 
            "max_size": "50GB" 
          } 
        } 
      }, 
      "delete": { 
        "min_age": "360d",            
        "actions": { 
          "delete": {}               
        } 
      } 
    } 
  } 
}

# Create the index template 
PUT _template/servicenow-incident-updates-template 
{ 
  "index_patterns": [ 
    "servicenow-incident-updates*" 
  ], 
  "settings": { 
    "number_of_shards": 1, 
    "index.lifecycle.name": "servicenow-incident-updates-policy",       
    "index.lifecycle.rollover_alias": "servicenow-incident-updates"     
  }, 
  "mappings": { 
    "properties": { 
      "@timestamp": { 
        "type": "date", 
        "format": "yyyy-MM-dd HH:mm:ss" 
      }, 
      "assignedTo": { 
        "type": "keyword" 
      }, 
      "description": { 
        "type": "text", 
        "fields": { 
          "keyword": { 
            "type": "keyword", 
            "ignore_above": 256 
          } 
        } 
      }, 
      "incidentID": { 
        "type": "keyword" 
      }, 
      "state": { 
        "type": "keyword" 
      }, 
      "app_name": { 
        "type": "keyword" 
      }, 
      "updatedDate": { 
        "type": "date", 
        "format": "yyyy-MM-dd HH:mm:ss" 
      }, 
      "workNotes": { 
        "type": "text" 
      } 
    } 
  } 
}

# Bootstrap the initial index and create the alias 
PUT servicenow-incident-updates-000001 
{ 
  "aliases": { 
    "servicenow-incident-updates": { 
      "is_write_index": true 
    } 
  } 
}

app_uptime_summary & app_incident_summary

As both of these indices are entity-centric, they do not need to have an ILM policy associated with them. This is because we will only ever have one document per application that we are monitoring. To create the indices, you issue the following commands:

PUT app_uptime_summary 
{ 
  "mappings": { 
    "properties": { 
      "hours_online": { 
        "type": "float" 
      }, 
      "app_name": { 
        "type": "keyword" 
      }, 
      "up_count": { 
        "type": "long" 
      }, 
      "last_updated": { 
        "type": "date" 
      } 
    } 
  } 
}

PUT app_incident_summary 
{ 
  "mappings": { 
    "properties" : { 
        "hours_online" : { 
          "type" : "float" 
        }, 
        "incident_count" : { 
          "type" : "integer" 
        }, 
        "app_name" : { 
           "type" : "keyword" 
        }, 
        "mtbf" : { 
          "type" : "float" 
        } 
      } 
  } 
}

Set up the two transforms

Transforms are an incredibly useful and recent addition to the Elastic Stack. They provide the capability to convert existing indices into an entity-centric summary, which is great for analytics and new insights. An often overlooked benefit of transforms is their performance benefits. For example, instead of trying to calculate the MTBF for each application by query and aggregation (which would get quite complicated), we can have a continuous transform calculating it for us on a cadence of our choice. For example, every minute! Without transforms they would be calculated once for every refresh that each person does on the Canvas workpad. Meaning, if we have 50 people using the workpad with a refresh interval of 30 seconds, we run the expensive query 100 times per minute (which seems a bit excessive). While this wouldn’t be an issue for Elasticsearch in most cases, I want to take advantage of this awesome new feature which makes life much easier.

We are going to create two transforms: 

• calculate_uptime_hours_online_transform: Calculates the number of hours each application has been online and responsive. It does this by utilizing the uptime data from Heartbeat. It will store these results in the app_uptime_summary index. 

• app_incident_summary_transform: combines the ServiceNow data with the uptime data coming from the previously mentioned transform (yes... sounds a bit like a join to me). This transform will take the uptime data and work out how many incidents each application has had, bring forward how many hours it has been online and finally calculate the MTBF based on those two metrics. The resulting index will be called app_incident_summary.

calculate_uptime_hours_online_transform

PUT _transform/calculate_uptime_hours_online_transform 
{ 
  "source": { 
    "index": [ 
      "heartbeat*" 
    ], 
    "query": { 
      "bool": { 
        "must": [ 
          { 
            "match_phrase": { 
              "monitor.status": "up" 
            } 
          } 
        ] 
      } 
    } 
  }, 
  "dest": { 
    "index": "app_uptime_summary" 
  }, 
  "sync": { 
    "time": { 
      "field": "@timestamp", 
      "delay": "60s" 
    } 
  }, 
  "pivot": { 
    "group_by": { 
      "app_name": { 
        "terms": { 
          "field": "monitor.name" 
        } 
      } 
    }, 
    "aggregations": { 
      "@timestamp": { 
        "max": { 
          "field": "@timestamp" 
        } 
      }, 
      "up_count": { 
        "value_count": { 
          "field": "monitor.status" 
        } 
      }, 
      "hours_online": { 
        "bucket_script": { 
          "buckets_path": { 
            "up_count": "up_count" 
          }, 
          "script": "(params.up_count * 60.0) / 3600.0" 
        } 
      } 
    } 
  }, 
  "description": "Calculate the hours online for each thing monitored by uptime" 
}

app_incident_summary_transform 

PUT _transform/app_incident_summary_transform 
{ 
  "source": { 
    "index": [ 
      "app_uptime_summary", 
      "servicenow*" 
    ] 
  }, 
  "pivot": { 
    "group_by": { 
      "app_name": { 
        "terms": { 
          "field": "app_name" 
        } 
      } 
    }, 
    "aggregations": { 
      "incident_count": { 
        "cardinality": { 
          "field": "incidentID" 
        } 
      }, 
      "hours_online": { 
        "max": { 
          "field": "hours_online", 
          "missing": 0 
        } 
      }, 
      "mtbf": { 
        "bucket_script": { 
          "buckets_path": { 
            "hours_online": "hours_online", 
            "incident_count": "incident_count" 
          }, 
          "script": "(float)params.hours_online / (float)params.incident_count" 
        } 
      } 
    } 
  }, 
  "description": "Calculates the MTBF for apps by using the output from the calculate_uptime_hours_online transform", 
  "dest": { 
    "index": "app_incident_summary" 
  }, 
  "sync": { 
    "time": { 
      "field": "@timestamp", 
      "delay": "1m" 
    } 
  } 
}

Let’s now ensure that both transforms are running:

POST _transform/calculate_uptime_hours_online_transform/_start 
POST _transform/app_incident_summary_transform/_start

Uptime alerts to create tickets in ServiceNow

Apart from making a pretty Canvas workpad, the final step to close the loop is to actually create a ticket in ServiceNow if one doesn't already exist for this outage. To do this, we are going to use Watcher to create an alert. This alert has the steps outlined below. For context, it runs every minute. You can see all the uptime fields in the Heartbeat documentation.

1. Check to see what applications have been down in the past 5 minutes

This is the simple bit. We are going to get all Heartbeat events which have been down (term filter) in the past 5 minutes (range filter) grouped by the monitor.name (terms aggregation). This monitor.name field is following Elastic Common Schema (ECS) and will be synonymous to the value in our application name field. This is all achieved through the down_check input in the watcher below.

2. Get the top twenty tickets for each application and get the latest update for each ticket

This trends the line of complexity. We are going to search across our ingested ServiceNow data which is automatically ingested due to our ServiceNow business rule. The existing_ticket_check input uses multiple aggregations. The first is to group all of the applications together through the “apps” terms aggregation. For each app we then group the ServiceNow ticket incident IDs together by using a terms aggregation called incidents. Finally, for each incident found for each application, we are going to get the latest state by using the top_hits aggregation ordered by the @timestamp field. 

3. Merge the two feeds together and see if any tickets need creating

To achieve this, we use script payload transform. In short, it checks to see what is down by iterating over the down_check output and then checks to see if that particular application has an open ticket. If there is not a ticket currently in progress, new or on hold, it adds the application to a list which is returned and passed onto the action phase. 

This payload transform does quite a few checks in the middle of this to catch the edge cases I have outlined below, such as creating a ticket if the app hasn’t had any incident history before. The output of this transform is an array of application names.

4. If new, create the ticket in ServiceNow

We use the webhook action to create the ticket in ServiceNow by using its REST API. To do this, it uses the foreach parameter to iterate over the application names from the above array and then runs the webook action for each of them. It will only do this if there is one or more applications that need a ticket. Please ensure that you set the correct credentials and endpoint for ServiceNow.

PUT _watcher/watch/e146d580-3de7-4a4c-a519-9598e47cbad1 
{ 
  "trigger": { 
    "schedule": { 
      "interval": "1m" 
    } 
  }, 
  "input": { 
    "chain": { 
      "inputs": [ 
        { 
          "down_check": { 
            "search": { 
              "request": { 
                "body": { 
                  "query": { 
                    "bool": { 
                      "filter": [ 
                        { 
                          "range": { 
                            "@timestamp": { 
                              "gte": "now-5m/m" 
                            } 
                          } 
                        }, 
                        { 
                          "term": { 
                            "monitor.status": "down" 
                          } 
                        } 
                      ] 
                    } 
                  }, 
                  "size": 0, 
                  "aggs": { 
                    "apps": { 
                      "terms": { 
                        "field": "monitor.name", 
                        "size": 100 
                      } 
                    } 
                  } 
                }, 
                "indices": [ 
                  "heartbeat-*" 
                ], 
                "rest_total_hits_as_int": true, 
                "search_type": "query_then_fetch" 
              } 
            } 
          } 
        }, 
        { 
          "existing_ticket_check": { 
            "search": { 
              "request": { 
                "body": { 
                  "aggs": { 
                    "apps": { 
                      "aggs": { 
                        "incidents": { 
                          "aggs": { 
                            "state": { 
                              "top_hits": { 
                                "_source": "state", 
                                "size": 1, 
                                "sort": [ 
                                  { 
                                    "@timestamp": { 
                                      "order": "desc" 
                                    } 
                                  } 
                                ] 
                              } 
                            } 
                          }, 
                          "terms": { 
                            "field": "incidentID", 
                            "order": { 
                              "_key": "desc" 
                            }, 
                            "size": 1 
                          } 
                        } 
                      }, 
                      "terms": { 
                        "field": "app_name", 
                        "size": 100 
                      } 
                    } 
                  }, 
                  "size": 0 
                }, 
                "indices": [ 
                  "servicenow*" 
                ], 
                "rest_total_hits_as_int": true, 
                "search_type": "query_then_fetch" 
              } 
            } 
          } 
        } 
      ] 
    } 
  }, 
  "transform": { 
   "script": """ 
      List appsNeedingTicket = new ArrayList();  
      for (app_heartbeat in ctx.payload.down_check.aggregations.apps.buckets) { 
        boolean appFound = false;  
        List appsWithTickets = ctx.payload.existing_ticket_check.aggregations.apps.buckets;  
        if (appsWithTickets.size() == 0) {  
          appsNeedingTicket.add(app_heartbeat.key);  
          continue;  
        }  
        for (app in appsWithTickets) {   
          boolean needsTicket = false;  
          if (app.key == app_heartbeat.key) {  
            appFound = true;  
            for (incident in app.incidents.buckets) {  
              String state = incident.state.hits.hits[0]._source.state;  
              if (state == 'Resolved' || state == 'Closed' || state == 'Canceled') {  
                appsNeedingTicket.add(app.key);  
              }  
            }  
          }  
        }  
        if (appFound == false) {  
          appsNeedingTicket.add(app_heartbeat.key);  
        }  
      }  
      return appsNeedingTicket;  
      """ 
  }, 
  "actions": { 
    "submit_servicenow_ticket": { 
      "condition": { 
        "script": { 
          "source": "return ctx.payload._value.size() > 0" 
        } 
      }, 
      "foreach": "ctx.payload._value", 
      "max_iterations": 500, 
      "webhook": { 
        "scheme": "https", 
        "host": "dev94721.service-now.com", 
        "port": 443, 
        "method": "post", 
        "path": "/api/now/table/incident", 
        "params": {}, 
        "headers": { 
          "Accept": "application/json", 
          "Content-Type": "application/json" 
        }, 
        "auth": { 
          "basic": { 
            "username": "admin", 
            "password": "REDACTED" 
          } 
        }, 
       "body": "{'description':'{{ctx.payload._value}} Offline','short_description':'{{ctx.payload._value}} Offline','caller_id': 'elastic_watcher','impact':'1','urgency':'1', 'u_application':'{{ctx.payload._value}}'}", 
        "read_timeout_millis": 30000 
      } 
    } 
  }, 
  "metadata": { 
    "xpack": { 
      "type": "json" 
    }, 
    "name": "ApplicationDowntime Watcher" 
  } 
}

Conclusion

This is where we conclude the second installment of this project. In this section we have created some general configuration for Elasticsearch so that the indices and ILM policy exist. Within this section we also created the Transforms which calculate the uptime and MTBF for each application as well as the Watcher which monitors our uptime data. The key thing to take note here is that, if the Watcher notices that something goes down it first checks if a ticket exists and if not creates one in ServiceNow.

Interested in following along? The easiest way to do that is to use Elastic Cloud. Either log into the Elastic Cloud console or sign up for a free 14-day trial. You can follow the above steps with your existing ServiceNow instance or spin up a personal developer instance.

Also, if you’re looking to search over ServiceNow data along with other sources such as GitHub, Google Drive, and more, Elastic Workplace Search has a prebuilt ServiceNow connector. Workplace Search provides a unified search experience for your teams, with relevant results across all your content sources. It’s also included in your Elastic Cloud trial.

At this point, everything is functional! However not what I would call visually appealing. So, in the third and final section of this project we will look at how you can use Canvas to create a great looking front-end to represent all of this data, as well as calculate the other metrics mentioned such as MTTA, MTTR and more.

See you soon!

For all of us at Elastic, community matters. Our users and contributors have helped to ensure that Elasticsearch, Kibana, Beats, and Logstash are more than just code — they are open source projects that people love to use and love to talk about!

Organizing or participating in a meetup group can be an incredibly rewarding way to contribute not only to the Elastic Community, but also to the community in which you live or work — whether you're looking to gain public speaking experience, wishing to network with like-minded folks in your area, or hoping to "level up" the local community's knowledge base. As an added bonus, you can earn points for the Elastic Contributor Program by organizing events!

In this blog post, we’ll cover a few things you’ll want to consider when organizing an Elastic meetup. 

First, you’ll first need to do a little bit of research on the tech user groups in your area. If there’s an Elastic user group near you, reach out to the community organizer or meetups@elastic.co to express your interest in organizing an event. If there isn’t an Elastic user group nearby, look for general tech user groups or Java user groups and reach out to their organizers. It’s a good idea to give yourself at least two months to put the event together and promote it.

Next, you’ll need to find a venue for your meetup. If you’re planning a virtual meetup, we have some good news: the Elastic Community team has a virtual venue ready for you to use! We’re happy to help you with the technical setup. For live events, we most often host meetups at either the event organizer’s or speaker’s workplace. Libraries, pubs, and coworking spaces have worked well for us in the past as well. Note that we don’t have a budget to pay any venue fees. Our organizer guide has more tips on selecting a venue.

After identifying a target date and securing a venue for the event, it’s time to find a speaker. It’s most effective to start within your own network. Do you have a colleague or friend who has a unique use case? Maybe you can even do a technical deep dive into a part of the Elastic Stack that you use yourself. Another good strategy for finding a speaker for a live, one-off event is to look at speakers from past events at other tech user groups in your area and reach out to them through Meetup, Twitter, or LinkedIn. If you’re organizing a virtual event or plan to host a “watch party,” you can find potential speakers all over the internet! Try searching on YouTube, Medium, GitHub, and SlideShare for folks who are creating Elastic video tutorials, writing articles about their use cases, or committing code to the Elastic GitHub repo. After you’ve found a potential speaker, be sure to send them our speaker guide.

Once you’ve identified a speaker, you’ll need to get a bio and abstract from them. With this in hand you’re ready to publish the event on our events platform and meetup.com! Reach out to us at meetups@elastic.co for support with entering these details. It’s a good idea to put together a rough agenda for the meetup at this stage. Capping speaker’s presentations at about 25 minutes each helps ensure that the audience is able to remain engaged. For a virtual meetup, we like to open the Zoom room about 5 minutes before the start time, and kick off the presentation at about 3 minutes past the start time to give folks time to arrive. Be sure to ask the speaker to include a few “warm up” questions for the audience ahead of their presentation. After the presentation is over, budget about 10 minutes for Q&A. At a live event things can vary. Usually we like to reserve the first 30 minutes for networking, arrivals, and food and drink. Then the organizer introduces the speakers, the presentation is given, and finally we leave room for about 30 minutes of networking and Q&A at the end.

Now that the event is published and the agenda set it’s time to promote it! Share on the event on social media and with your network. The more you’re able to spread the word, the higher the attendance will be. 

If you’re hosting a live event or a live watch party, you’ll need to order food and drink for the event. We provide a budget for this and are happy to help you place the order. You’ll need to provide the delivery address and your phone number.

About a week before the event it’s a good idea to check in with the speaker and run through the meetup’s agenda. Have the speaker send you a copy of their slides just in case they’re running late to the event. On the day of, it’s a good idea to show up at least 15 minutes early to set up the food and drink if it’s a live event, or to test out the virtual environment if it’s a virtual event. During the event you’ll play the role of MC: introduce the speaker and make announcements to the group, including a reminder that attendees are expected to comply with our Community Code of Conduct. Introduce yourself to attendees as the organizer and help attendees network with one another. Once the presentation has ended, help the speaker field questions from attendees and thank everyone for coming!

After the event, send a recording (if you have one) along with a brief report (how many people attended, was there enough food?) to meetups@elastic.co. Don’t forget to sign up for the Elastic Contributor Program and submit the event you organized to earn points!

We hope these steps help you get started organizing your first Elastic meetup. Our community is part of what makes Elastic unique and we are truly grateful for your participation! If you have any questions along the way, please reach out to us at meetups@elastic.co.

Elastic held its first-ever fully virtual ElasticON Global event, featuring a public sector day, 19 keynotes, 300 breakout sessions, and 31 Ask Me Anything (AMA) and collaboration sessions from October 13-15, 2020.

ElasticON Global 2020 had something for everyone. Attendees learned how Elastic solutions help customers drive outcomes and business performance with data, insight, and action, and saw a preview of the Elastic vision and development direction for its solution areas across enterprise search, observability, and security, with demos of upcoming major product features.

Elastic CEO Shay Banon delivered the opening keynote debuting new solution innovations and discussed how those solutions harness the power of search to give users the fastest path from data to insight to action.

Keynote highlights include:

• A conversation with Google Cloud CEO Thomas Kurian, who joined Shay to share how the partnership between Elastic and Google Cloud helps accelerate business innovation for joint customers with Elastic’s free and paid proprietary features and solutions.
• Demos of upcoming product innovations, including Elastic Agent, Kibana Lens, actionable Kibana, searchable snapshots, and schema on read.
• Focused keynotes from Elastic leaders on Elastic Cloud, Elastic Stack and the Elastic Enterprise Search, Observability, and Security solutions.
• Fireside chats with customers about the KPIs that teams are tracking, the ways they’re using Elastic, critical features and capabilities, and more. The customers included Wells Fargo and Cerner in the Americas, Shell International in Europe, and international law firm King & Wood Mallesons in Asia-Pacific.

Public Sector Programming

The dedicated public sector day during ElasticON Global featured international government and higher education content.

Director of Solutions Architecture Dave Erickson shared the vision for the future of Elastic solutions for the public sector. Erickson presented new, impactful features, discussed GovCloud availability and FedRAMP authorization, and announced a new program for veterans called Operation Giving Back, which allows Elastic to make training and career advancement resources available to the veteran and active duty military community through a partnership with VetsinTech.

The public sector Elastic Search Awards were announced and the honorees are:

• Leidos, who built a data discovery platform making petabytes of sensitive mission data searchable and available to U.S. military and intelligence agencies.
• The UK National Crime Agency, who is using the Elastic Stack to enhance operational activity to cut both serious and organized crime.
• Stormfish Scientific Corporation, whose team is using Elastic to better understand how virtual and augmented reality impacts and empowers mission-critical decision making.

In addition, sessions featuring real-world government and higher education users sharing how Elastic is powering their cutting-edge use cases drove interest and engagement:

• Protecting against cyber attacks at UC Davis with Elastic
• Centralized logging in a changing environment at the UK’s DVLA
• How CACI and Elastic support the Department of Defense

Elastic General Manager Nate Fick closed the Public Sector event with a view of the cybersecurity universe today and the core philosophies that drive how Elastic builds security solutions to help customers meet today’s challenges. Nate discussed how attendees can stay agile, adapt, and protect what matters most with Elastic’s best-of-breed security solutions, in a world where mean time to detect (MTTD) still hovers at around 90 days and the barriers to entry for cyberattackers continue to fall.

CXO Programming

The event’s 300+ sessions offered insights for all levels of attendees and provided a special component created specifically for executive-level attendees. Forrester Research analyst Noel Yuhanna delivered a session about leveraging the cloud to turn data into actionable insights. Ravi Ravishankar, vice president product marketing, moderated a panel discussion with Noel, John Fair, CTO, Rocket Homes, and Neil Cains, CTO, Innovation Lab, Allegis Group that was curated specifically for event participants in C-level and management roles.

In addition, other sessions featured customer speakers who shared critical insights for managers and executives as they talked about their data challenges and subsequent triumphs in using Elastic, including:

• Innovation development subsidiary of German automaker Audi, Audi Business Innovation, who spoke about delivering observability solutions as a service at speed and scale to all developers in the VW Group using Elastic
• French financial institution Société Générale, who use Elastic for observability to bridge their technical and functional vision for their monitoring solution to create more business value
• And Cisco, who uses Elastic to deliver sub-second response times across billions of assets to the consumers of its analytics and insights platform for product entitlements

Other Event Highlights

• Microsoft Azure head of product, Microsoft Azure development tools and diagnostics Paul Yuknewicz shared best practices for running Elasticsearch Service on Microsoft Azure and how the partnership between Microsoft and Elastic makes it easier to deploy powerful search experiences in an Azure environment.
• Elastic Chief Marketing Officer Sally Jenkins had a fireside chat with soccer star and World Cup champion Megan Rapinoe and Elastic CEO Shay Banon hosted a conversation with comedian and host of The Daily Show Trevor Noah. Both discussions highlighted views on equality, diversity, determination, resiliency, and more.
• Elastic Senior Inclusion Officer Karen Penn led a panel discussion about power and privilege in technology where participants reflected on their journeys as leaders in technology and shared how they have overcome institutional biases and hurdles. The panelists included:
  • Madhura Chopda, Elastic Director of Engineering
  • Heather Savory, Non-Executive Director and Business Advisor at UK Parliament’s Information Authority.
  • Anjuan Simmons, Help Scout Engineering Coach
  • Gary Smyth Elastic, Vice President of Cloud Inside Sales

With more than 300 sessions now available, viewers can access compelling content and hear thoughtful insights about using data to uncover insights that drive action for all - from practitioners to CXOs.

The full lineup of ElasticON Global 2020 event sessions is now available on demand for anyone, in any global region. Take and look to learn more about solution features, technical sessions or customer use cases, and sign up for updates about upcoming, dedicated solution-focused events for continued deep dives.

Thanks to the ElasticON Global 2020 sponsors: Google Cloud, Microsoft Azure, Dell Technologies, Novetta, Pure Storage, Siemplify, Siren, SVA, Zebrium, Corelight, Cribl, Norseman Defense Technologies, and OverOps.