keycloak/docs/guides/server/logging.adoc

<#import "/templates/guide.adoc" as tmpl>
<#import "/templates/kc.adoc" as kc>
<#import "/templates/links.adoc" as links>

<@tmpl.guide
title="Configuring logging"
summary="Learn how to configure Logging"
includedOptions="log-*">

Keycloak uses the JBoss Logging framework. The following is a high-level overview for the available log handlers:

* root
** console (_default_)
** file
** GELF

== Logging configuration
Logging is done on a per-category basis in Keycloak. You can configure logging for the root log level or for more specific categories such as `org.hibernate` or `org.keycloak`. This guide describes how to configure logging.

=== Log levels

The following table defines the available log levels.

|====
|Level|Description
|FATAL|Critical failures with complete inability to serve any kind of request.
|ERROR|A significant error or problem leading to the inability to process requests.
|WARN|A non-critical error or problem that might not require immediate correction.
|INFO|Keycloak lifecycle events or important information. Low frequency.
|DEBUG|More detailed information for debugging purposes, such as database logs. Higher frequency.
|TRACE|Most detailed debugging information. Very high frequency.
|ALL|Special level for all log messages.
|OFF|Special level to turn logging off entirely (not recommended).
|====

=== Configuring the root log level
When no log level configuration exists for a more specific category logger, the enclosing category is used instead. When there is no enclosing category, the root logger level is used.

To set the root log level, enter the following command:

<@kc.start parameters="--log-level=<root-level>"/>

Use these guidelines for this command:

* For `_<root-level>_`, supply a level defined in the preceding table.
* The log level is case-insensitive. For example, you could either use `DEBUG` or `debug`.
* If you were to accidentally set the log level twice, the last occurrence in the list becomes the log level. For example, if you included the syntax `--log-level="info,...,DEBUG,..."`, the root logger would be `DEBUG`.

=== Configuring category-specific log levels
You can set different log levels for specific areas in Keycloak. Use this command to provide a comma-separated list of categories for which you want a different log level:

<@kc.start parameters="--log-level=\"<root-level>,<org.category1>:<org.category1-level>\""/>

A configuration that applies to a category also applies to its sub-categories unless you include a more specific matching sub-category.

.Example
<@kc.start parameters="--log-level=\"INFO,org.hibernate:debug,org.hibernate.hql.internal.ast:info\""/>

This example sets the following log levels:

* Root log level for all loggers is set to INFO.
* The hibernate log level in general is set to debug.
* To keep SQL abstract syntax trees from creating verbose log output, the specific subcategory `org.hibernate.hql.internal.ast` is set to info. As a result, the SQL abstract syntax trees are omitted instead of appearing at the `debug` level.

== Enabling log handlers
To enable log handlers, enter the following command:

<@kc.start parameters="--log=\"<handler1>,<handler2>\""/>

The available handlers are `console`, `file` and `gelf`. The more specific handler configuration mentioned below will only take effect when the handler is added to this comma-separated list.

== Console log handler
The console log handler is enabled by default, providing unstructured log messages for the console.

=== Configuring the console log format
Keycloak uses a pattern-based logging formatter that generates human-readable text logs by default.

The logging format template for these lines can be applied at the root level. The default format template is:

* `%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n`

The format string supports the symbols in the following table:

|====
|Symbol|Summary|Description
|%%|%|Renders a simple % character.
|%c|Category|Renders the log category name.
|++%d{xxx}++|Date|Renders a date with the given date format string.String syntax defined by `java.text.SimpleDateFormat`
|%e|Exception|Renders a thrown exception.
|%h|Hostname|Renders the simple host name.
|%H|Qualified host name|Renders the fully qualified hostname, which may be the same as the simple host name, depending on the OS configuration.
|%i|Process ID|Renders the current process PID.
|%m|Full Message|Renders the log message and an exception, if thrown.
|%n |Newline|Renders the platform-specific line separator string.
|%N|Process name|Renders the name of the current process.
|%p|Level|Renders the log level of the message.
|%r|Relative time|Render the time in milliseconds since the start of the application log.
|%s|Simple message|Renders only the log message without exception trace.
|%t|Thread name|Renders the thread name.
|%t{id}|Thread ID|Render the thread ID.
|%z{<zone name>}|Timezone|Set the time zone of log output to <zone name>.
|%L|Line number|Render the line number of the log message.
|====

=== Setting the logging format
To set the logging format for a logged line, perform these steps:

. Build your desired format template using the preceding table.
. Enter the following command:
+
<@kc.start parameters="--log-console-format=\"\'<format>\'\""/>

Note that you need to escape characters when invoking commands containing special shell characters such as `;` using the CLI. Therefore, consider setting it in the configuration file instead.

.Example: Abbreviate the fully qualified category name
<@kc.start parameters="--log-console-format=\"\'%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) %s%e%n\'\""/>

This example abbreviates the category name to three characters by setting `[%c{3.}]` in the template instead of the default `[%c]`.

=== Configuring JSON or plain console logging
By default, the console log handler logs plain unstructured data to the console. To use structured JSON log output instead, enter the following command:

<@kc.start parameters="--log-console-output=json"/>

.Example Log Message
[source, json]
----
{"timestamp":"2022-02-25T10:31:32.452+01:00","sequence":8442,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus","level":"INFO","message":"Keycloak 18.0.0-SNAPSHOT on JVM (powered by Quarkus 2.7.2.Final) started in 3.253s. Listening on: http://0.0.0.0:8080","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"host-name","processName":"QuarkusEntryPoint","processId":36946}
----

When using JSON output, colors are disabled and the format settings set by `--log-console-format` will not apply.

To use unstructured logging, enter the following command:

<@kc.start parameters="--log-console-output=default"/>

.Example Log Message:
[source, bash]
----
2022-03-02 10:36:50,603 INFO  [io.quarkus] (main) Keycloak 18.0.0-SNAPSHOT on JVM (powered by Quarkus 2.7.2.Final) started in 3.615s. Listening on: http://0.0.0.0:8080
----

=== Colors
Colored console log output for unstructured logs is disabled by default. Colors may improve readability, but they can cause problems when shipping logs to external log aggregation systems. To enable or disable color-coded console log output, enter following command:

<@kc.start parameters="--log-console-color=<false|true>"/>

== File logging
As an alternative to logging to the console, you can use unstructured logging to a file.

=== Enable file logging
Logging to a file is disabled by default. To enable it, enter the following command:

<@kc.start parameters="--log=\"console,file\""/>

A log file named `keycloak.log` is created inside the `data/log` directory of your Keycloak installation.

=== Configuring the location and name of the log file

To change where the log file is created and the file name, perform these steps:

. Create a writable directory to store the log file.
+
If the directory is not writable, Keycloak will start correctly, but it will issue an error and no log file will be created.

. Enter this command:
+
<@kc.start parameters="--log=\"console,file\" --log-file=<path-to>/<your-file.log>"/>

=== Configuring the file handler format
To configure a different logging format for the file log handler, enter the following command:

<@kc.start parameters="--log-file-format=\"<pattern>\""/>

See <<Configuring the console log format>> for more information and a table of the available pattern configuration.

== Centralized logging using GELF
Keycloak can send logs to a centralized log management system such as the following:

* Graylog
* Logstash, inside the Elasticsearch, Logstash, Kibana (ELK) logging stack
* Fluentd, inside the Elasticsearch, Fluentd, Kibana (EFK) logging stack

Keycloak uses the https://quarkus.io/guides/centralized-log-management[Quarkus Logging GELF] extension to support these environments.

=== Enabling the GELF handler
To enable logging using GELF, add it to the list of activated log handlers.

.Example:
<@kc.start parameters="--log=\"console,gelf\""/>

=== Configuring the GELF handler

To configure the Host and Port of your centralized logging system, enter the following command and substitute the values with your specific values:
.Host and port of the GELF server:
<@kc.start parameters="--log=\"console,gelf\" --log-gelf-host=myhost --log-gelf-port=12345"/>

When the GELF handler is enabled, the host is using `localhost` as host value and UDP for communication. To use TCP instead of UDP, prefix the host value with `tcp:`. The Default port is `12201`.

.Include or exclude Stacktraces
Keycloak includes the complete Stacktrace inside the `StackTrace` field. To exclude this field, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-stack-trace=false"/>

.Configure the timestamp format
You can change the format of the `timestamp` field. For example, you can include the date and time down to seconds by entering the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-timestamp-format=\"\'yyyy-MM-dd HH:mm:ss\'\""/>

Alternatively, you could use the config file to avoid escaping:

[source, conf]
----
log-gelf-timestamp-format=yyyy-MM-dd HH:mm:ss
----

The default timestamp format is `yyyy-MM-dd HH:mm:ss,SSS`. You can use the https://docs.oracle.com/javase/10/docs/api/java/text/SimpleDateFormat.html[available SimpleDateFormat patterns] to define an appropriate timestamp.

.Configure the facility
The `facility` field is an indicator of the process or program that is the source of log messages. The default value is `keycloak`. To set this field to your preferred identifier, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-facility=MyKeycloak"/>

To use the CLI to configure Keycloak and use whitespaces for `facility`, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-facility=\"\'my keycloak\'\""/>

Alternatively, you could use the config file to avoid escaping:

[source, conf]
----
log-gelf-facility=my keycloak
----

.Configure the default message size
To change the default message size of 8kb (8192 bytes) of GELF log messages for Keycloak, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-max-message-size=16384"/>

The maximum size of one GELF log message is set in Bytes. The preceding example increases the size to 16kb. When messages exceed the maximum size, GELF submits the message in multiple chunks.

.Configure sending of message parameters
Keycloak includes message parameters of the occurred log event. These fields appear in the output as `MessageParam0`, `MessageParam1`, and so on, depending on the parameter length.
To switch off this behavior, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-message-parameters=false"/>

.Configure sending of source code location
Keycloak includes the `SourceClassName`, `SourceMethodName` and `SourceSimpleClassName` fields in the GELF log messages. These fields provide detail on the location of an exception that occurred. To stop sending these fields, enter the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-gelf-include-location=false"/>

=== Example: Send logs to Graylog
The following example shows how to send Keycloak logs to the Graylog centralized logging stack. This example assumes you have a container tool such as https://www.docker.com/[docker] installed to start the `compose.yml`.

==== Starting the Graylog stack
The composed stack consists of:

* Graylog
* ElasticSearch
* MongoDB

[source, yaml]
----
version: '3.8'

services:
  elasticsearch:
    image: docker.io/elastic/elasticsearch:7.10.2
    ports:
      - "9200:9200"
    environment:
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
      discovery.type: "single-node"
    networks:
      - graylog

  mongo:
    image: mongo:4.4
    networks:
      - graylog

  graylog:
    image: graylog/graylog:4.3.3
    ports:
      - "9000:9000"
      - "12201:12201/udp"
      - "1514:1514"
    environment:
      GRAYLOG_HTTP_EXTERNAL_URI: "http://127.0.0.1:9000/"
      # CHANGE ME (must be at least 16 characters)!
      GRAYLOG_PASSWORD_SECRET: "forpasswordencryption"
      # Password: admin
      GRAYLOG_ROOT_PASSWORD_SHA2: "8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918"
    networks:
      - graylog
    depends_on:
      - elasticsearch
      - mongo

networks:
  graylog:
    driver: bridge
----

Copy and save the example locally into a `compose.yml` file and enter this command:

[source,bash]
----
docker compose up -d
----
After a few seconds, the Stack is ready to serve requests.

==== Creating a Graylog UDP Input
Once the stack is running, you need to create a UDP Input Graylog listens to. You can create it from the Graylog web UI (System → Input → Select GELF UDP) available at http://localhost:9000 or using the API:

This `curl` example creates a new GELF UDP Input using the API and the default Graylog login credentials (admin/admin).

[source, bash]
----
curl -H "Content-Type: application/json" -H "Authorization: Basic YWRtaW46YWRtaW4=" -H "X-Requested-By: curl" -X POST -v -d \
'{"title":"udp input","configuration":{"recv_buffer_size":262144,"bind_address":"0.0.0.0","port":12201,"decompress_size_limit":8388608},"type":"org.graylog2.inputs.gelf.udp.GELFUDPInput","global":true}' \
http://localhost:9000/api/system/inputs
----

If the stack is still in the bootstrap phase, you receive a response containing `* Empty reply from server`. A successful response includes `HTTP/1.1 201 Created` to indicate that the UDP input is created.

==== Configure Keycloak to send logs using GELF
Keycloak needs to be configured to send logs using GELF. The appropriate configuration can be seen in the following keycloak.conf example. The example includes the `log-gelf-host` and `log-gelf-port` values. These are optional values that are included for illustration purposes; default values exist.

.Keycloak GELF Configuration

[source, conf]
----
log=console,gelf
log-gelf-host=localhost
log-gelf-port=12201
----

==== Graylog: See the results
. Open your web browser, go to `http://localhost:9000`.
. Log in to the Graylog web UI using the administrator credentials (admin/admin).
. Go to Streams, All Messages.
. Start updating the stream by pressing the Play button in the upper right corner.
. Start Keycloak using `start` or `start-dev` and your GELF config.

After a few seconds, Keycloak messages appear in the Graylog dashboard.

=== Example Setup using the ELK Stack
The following example shows how to send Keycloak logs to the ELK centralized logging stack. It assumes you have a container tool such as https://www.docker.com/[docker] installed to start the `compose.yml`.

==== Enable the logstash GELF plugin and create a pipeline
Logstash uses an input plugin that understands and parses the GELF format. To activate this plugin when you are starting the ELK stack later on, create a directory `pipelines` and a file `gelf.conf` located in this directory. Then create an empty `compose.yml` in the parent directory.

.File Structure:
[source]
----
/ELK
  - compose.yml
  - pipelines/
    - gelf.conf
----


Add the following contents to `pipelines/gelf.conf` and save it:

[source, conf]
----
input {
  gelf {
    port => 12201
  }
}
output {
  stdout {}
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
  }
}
----

This file activates and configures the logstash GELF plugin and points it to the right elasticsearch instance.

==== Starting the ELK stack
The composed stack consists of:

* ElasticSearch
* Logstash
* Kibana

Copy the following content to your `compose.yml` file:

[source, yaml]
----
# Launch Elasticsearch
version: '3.8'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash-oss:6.8.2
    volumes:
      - source: ./pipelines #the source dir gelf.conf resides
        target: /usr/share/logstash/pipeline
        type: bind
    ports:
      - "12201:12201/udp"
      - "5000:5000"
      - "9600:9600"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana-oss:6.8.2
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    driver: bridge
----
Start the stack by entering the following command:

[source, bash]
----
docker compose up -d
----
After a few seconds the Stack should be ready to serve requests.

==== Configuring Keycloak to send logs using GELF
Keycloak needs to be configured to send logs using GELF. The appropriate configuration can be seen in the following keycloak.conf example. This example includes the `log-gelf-host` and `log-gelf-port` values. These are optional values, which are included for illustration purposes; default values exist.

.Keycloak Gelf Configuration

[source, conf]
----
log=console,gelf
log-gelf-host=localhost
log-gelf-port=12201
----

With this configuration applied, start keycloak using `start-dev` or `start`.

==== Kibana: See the results
Open http://localhost:5601 to reach the Kibana dashboard. The exact configuration of a good monitoring dashboard is out of scope for this guide. To find out if logs sent by Keycloak are delivered to Kibana, open the http://localhost:5601/app/kibana#/dev_tools/console?_g=()[Dev Tools] and execute the default `match_all` query. The logs should appear in the result field.

=== Configure a different log level for the GELF logger
To keep log storage costs and verbosity low, it is often wanted to only store a subset of the verbose application logs inside a centralized log management system. To configure Keycloak to use a different log level for the logs you want to ingest, use the following configuration:

[source, conf]
----
log=console,gelf
log-gelf-level=<desired-log-level>
...
----

.Example
To only see occurred log levels of warn and above in your centralized logging stack, but still see INFO level logs on the applications console logs, use the following configuration:

[source, conf]
----
log=console,gelf
log-level=INFO
log-gelf-level=warn
...
----

Looking at your ingested logs, you will see only messages of level warn or above will appear.

Keep in mind that `--log-level` is setting the leading log level, so for example when you invoke the following command:

<@kc.start parameters="--log=\"console,gelf\" --log-level=error --log-gelf-level=info"/>

nothing below the error level will be sent to your logging stack. That means that even GELF in this example will receive only error level log messages.

=== Configure additional key values
Currently, the Keycloak configuration does not support partly dynamic configuration keys, as they are used in quarkus properties. For example, they are used when defining `quarkus.log.handler.gelf.additional-field.<my-name>.value`.

To add user-defined fields, you can provide these fields through a quarkus.properties file. See <@links.server id="configuration"/> and the _Using raw Quarkus properties_ section.

</@tmpl.guide>