Elasticsearch geoip file issue

Good morning everyone,

It’s been a long while for a new more technical post.  Here’s a problem I encountered upon upgrading my Elasticsearch main server from 7.x to 8.12. I used the .deb from Elastic and everything seemed to go well for Elasticsearch and Kibana.

When checking ingested packet logs from Zeek I noticed the following in the ‘tags’ column:

[zeek.ssl, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-City.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb, _geoip_database_unavailable_GeoLite2-ASN.mmdb]

With the usual geoip data missing from everything except my local network.

In the elasticsearch logs the following popped up:

[ERROR][o.e.i.g.GeoIpDownloader ] [delphi] exception during geoip databases update
org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active

[WARN ][o.e.i.g.GeoIpDownloader ] [delphi] could not delete old chunks for geoip database [GeoLite2-City.mmdb]

After doing some research, I found a support entry regarding the geoip processor.  Essentially, the geoip processor automatically starts on elasticsearch on each restart, but it doesn’t clear the database if it sees a problem or corruption.

How to fix?

In your elasticsearch.yml file add the following:

ingest.geoip.downloader.enabled: false

Save the file and restart elasticsearch.  You should see in the elasticsearch log:

[o.e.c.m.MetadataDeleteIndexService] [delphi] [.geoip_databases/gOSLTuHvQSWsN2jUy1SfMg] deleting index

Go back into your elasticsearch.yml and modify:

ingest.geoip.downloader.enabled: true

Save and restart the elasticsearch service.  You then should see the geoip database indices download and integrate into your logs.

[INFO ][o.e.c.m.MetadataCreateIndexService] [delphi] [.geoip_databases] creating index, cause [auto(bulk api)], templates [], shards [1]/[0]

[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-ASN.mmdb]
[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-City.mmdb]
[INFO ][o.e.i.g.GeoIpDownloader ] [delphi] successfully downloaded geoip database [GeoLite2-Country.mmdb]
[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-Country.mmdb]
[INFO ][o.e.i.g.DatabaseNodeService] [delphi] successfully loaded geoip database file [GeoLite2-City.mmdb]

Hope this helps!