{"id":1517,"date":"2024-03-15T07:29:05","date_gmt":"2024-03-15T12:29:05","guid":{"rendered":"https:\/\/secognition.com\/?p=1517"},"modified":"2024-03-15T07:30:53","modified_gmt":"2024-03-15T12:30:53","slug":"elasticsearch-geoip-file-issue","status":"publish","type":"post","link":"https:\/\/secognition.com\/?p=1517","title":{"rendered":"Elasticsearch geoip file issue"},"content":{"rendered":"<p>Good morning everyone,<\/p>\n<p>It&#8217;s been a long while for a new more technical post.\u00a0 Here&#8217;s a problem I encountered upon upgrading my Elasticsearch main server from 7.x to 8.12. I used the .deb from Elastic and everything seemed to go well for Elasticsearch and Kibana.<\/p>\n<p>When checking ingested packet logs from Zeek I noticed the following in the &#8216;tags&#8217; column:<\/p>\n<div class=\"euiTableCellContent kbnDocViewer__tableValueCell euiTableCellContent--overflowingContent\">\n<pre class=\"kbnDocViewer__value\" data-test-subj=\"tableDocViewRow-tags-value\"><span class=\"ffArray__highlight\">[<\/span>zeek.ssl<span class=\"ffArray__highlight\">,<\/span> _geoip_database_unavailable_GeoLite2-City.mmdb<span class=\"ffArray__highlight\">,<\/span> _geoip_database_unavailable_GeoLite2-City.mmdb<span class=\"ffArray__highlight\">,<\/span> _geoip_database_unavailable_GeoLite2-ASN.mmdb<span class=\"ffArray__highlight\">,<\/span> _geoip_database_unavailable_GeoLite2-ASN.mmdb<span class=\"ffArray__highlight\">]<\/span><\/pre>\n<\/div>\n<p>With the usual geoip data missing from everything except my local network.<\/p>\n<p>In the elasticsearch logs the following popped up:<\/p>\n<pre>[ERROR][o.e.i.g.GeoIpDownloader ] [delphi] exception during geoip databases update\r\norg.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active\r\n\r\n[WARN ][o.e.i.g.GeoIpDownloader ] [delphi] could not delete old chunks for geoip database [GeoLite2-City.mmdb]<\/pre>\n<p>After doing some research, I found a support entry regarding the geoip processor.\u00a0 Essentially, the geoip processor automatically starts on elasticsearch on each restart, but it doesn&#8217;t clear the database if it sees a problem or corruption.<\/p>\n<p>&nbsp;<\/p>\n<p><strong>How to fix?<\/strong><\/p>\n<p>In your elasticsearch.yml file add the following:<\/p>\n<pre>ingest.geoip.downloader.enabled: false<\/pre>\n<p>Save the file and restart elasticsearch.\u00a0 You should see in the elasticsearch log:<\/p>\n<pre>[o.e.c.m.MetadataDeleteIndexService] [delphi] [.geoip_databases\/gOSLTuHvQSWsN2jUy1SfMg] deleting index<\/pre>\n<p>&nbsp;<\/p>\n<p>Go back into your elasticsearch.yml and modify:<\/p>\n<pre>ingest.geoip.downloader.enabled: true<\/pre>\n<p>Save and restart the elasticsearch service.\u00a0 You then should see the geoip database indices download and integrate into your logs.<\/p>\n<pre>[INFO ][o.e.c.m.MetadataCreateIndexService] [delphi] [.geoip_databases] creating index, cause [auto(bulk api)], templates [], shards [1]\/[0]\r\n\r\n[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-ASN.mmdb]\r\n[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-ASN.mmdb]\r\n[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-City.mmdb]\r\n[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-Country.mmdb]\r\n[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-Country.mmdb]\r\n[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-City.mmdb]<\/pre>\n<p>Hope this helps!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Good morning everyone, It&#8217;s been a long while for a new more technical post.\u00a0 Here&#8217;s a problem I encountered upon upgrading my Elasticsearch main server from 7.x to 8.12. I used the .deb from Elastic and everything seemed to go well for Elasticsearch and Kibana. When checking ingested packet logs from Zeek I noticed the&hellip; <br \/> <a class=\"read-more\" href=\"https:\/\/secognition.com\/?p=1517\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1517","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/secognition.com\/index.php?rest_route=\/wp\/v2\/posts\/1517","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/secognition.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/secognition.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/secognition.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/secognition.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1517"}],"version-history":[{"count":0,"href":"https:\/\/secognition.com\/index.php?rest_route=\/wp\/v2\/posts\/1517\/revisions"}],"wp:attachment":[{"href":"https:\/\/secognition.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1517"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/secognition.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1517"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/secognition.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1517"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}