diff --git a/modules/troubleshooting/pages/log-files.adoc b/modules/troubleshooting/pages/log-files.adoc index f79217009..3817afd8c 100644 --- a/modules/troubleshooting/pages/log-files.adoc +++ b/modules/troubleshooting/pages/log-files.adoc @@ -1,24 +1,30 @@ = Log Files -The TigerGraph database captures key information on activities occurring across its different components through log functions that output to log files. -These log files are not only helpful in xref:troubleshooting-guide.adoc[troubleshooting] but also serve as a resource for auditing. +TigerGraph captures key information about activities across its components through log files. These logs are essential for xref:troubleshooting:troubleshooting-guide.adoc[troubleshooting] and auditing. Logs may contain sensitive information, so direct access is restricted. -This page provides a general overview of the way log files are stored in TigerGraph. - +This page provides an overview of the log files available in TigerGraph, including where to find them, how they are stored, and what information they contain. xref:audit-log.adoc[Audit logs] are structured in JSON, ensuring machine-readability and facilitating easy integration with third-party tools. -The TigerGraph Linux server admin user may also use the xref:gcollect.adoc[gcollect] utility to search for and gather selected information from the logs. +TigerGraph Linux admin users also use the xref:gcollect.adoc[gcollect] utility to search for and gather selected information from the logs. We also provide instructions on how to xref:elk-filebeat.adoc[set up log viewing with Elasticsearch, Kibana, or Filebeat]. +== Available Log Files -== TigerGraph log structure +TigerGraph generates a variety of log files for its different components. +Understanding what logs are available and what they contain is the first step in effective troubleshooting and system monitoring. -Logs in TigerGraph are stored in TigerGraph's log root directory, which is configured at install time. -You can find the location by running the console command `gadmin config get System.LogRoot`. +=== Log File Locations -Within this directory are separate directories for the various TigerGraph services: +Logs in TigerGraph are stored in the log root directory, which is configured at install time. You can find this location by running: + +[source,console] +---- +gadmin config get System.LogRoot +---- + +Within this directory, you will find subdirectories for each TigerGraph component (admin, gpe, gsql, gui, kafka, nginx, zk, etc.). [source,console] ---- @@ -27,7 +33,7 @@ admin dict executor gpe gsql informant kafkaconn nginx zk controller etcd fileLoader gse gui kafka kafkastrm-ll restpp ---- -You can also use the `gadmin log` command to list log files: +Use the `gadmin log` command to list log files: [source, console] ---- @@ -41,7 +47,7 @@ ZK : /home/tigergraph/tigergraph/log/zk/ZK#1.out ZK : /home/tigergraph/tigergraph/log/zk/zookeeper.log ---- -Use the command `gadmin log ` to just get the logs for a specific service: +Use the command `gadmin log ` to get the logs for a specific service: [source, console] ---- @@ -50,34 +56,88 @@ GPE : /home/tigergraph/tigergraph/log/gpe/GPE_1#1.out GPE : /home/tigergraph/tigergraph/log/gpe/log.INFO ---- -The `log.INFO` file contains messages logged by the application code. -The `.out` log contains the redirection of the process output, and is used for debugging significantly less frequently than `log.INFO`. +Third-Party components like Zookeeper and Kafka have logs that are not listed by `gadmin log`. You can find them at: -[CAUTION] -The log format differs between the `.out` and `INFO` logs. -It also differs between certain TigerGraph services. -An internal project to unify log formats is ongoing. +[source,console] +---- +zookeeper : ~/tigergraph/zk/zookeeper.out.* +kafka : ~/tigergraph/kafka/kafka.out +---- + +=== TigerGraph Component Log Files + +* `.out` files capture *standard output (stdout)* and log runtime information, including error stack traces when services crash or unexpected errors occur. +These logs are especially useful for errors that aren't logged by the service's internal logging mechanism. + +* `.ERROR` files are used to log errors captured by the system, typically from exceptions caught in try-catch blocks. If an error occurs before the logging system initializes or is uncaught, it is logged in the `.out` file instead. + +* `.INFO` files log regular operational information about the system's normal functioning. + +To diagnose an issue for a given component, check the `.out` log file for that component. + +image::https://lh5.googleusercontent.com/6MnNakec5fKh5faCoWdZwfzprqXyguDZXt15nz0QAG1M3vW1t0nmwo7oYr3DgwVsgJoIEjub-5tSA81UtOQ-Ot-9m30zZ9Zr5tRG077dgfZ7KaE3tMMafUK63oi6fILQeM-kQw6fKqc[] + +[NOTE] +==== +The GUI component writes all log levels to a single log file and does not generate separate `.log`, `.error`, or `.info` files. + +* Each GUI log file (e.g., `gui_ADMIN.log`, `gui_INFO.log`) captures the standard output of the GUI process and includes all log levels (error, warning, info). +* The log level for the GUI component is *configurable*. You can set it using: + +[source,console] +---- +gadmin config set GUI.BasicConfig.LogConfig.LogLevel +---- + +Replace `` with one of: `DEBUG`, `INFO`, `WARN`, `ERROR`, `PANIC`, or `FATAL`. The default level is `INFO`. +==== + +==== Symbolic Links + +In directories with frequently checked logs, such as `restpp`, `gsql`, and `admin`, symbolic links make it easier to access the latest log file. +These links are automatically updated to point to the newest log. + +For example, `log.INFO` is a symbolic link that points to the current `.INFO` log file. To see what a symbolic link points to, use `ls -ll` followed by the symbolic link name: + +[source,console] +---- +ls -ll log.INFO +log.INFO -> log.INFO.2024-07-01-10-00-00 +---- + +Here, `log.INFO` is a symbolic link pointing to the current `.INFO` log file. -Log formats also vary across the different components. -In folders where logs are checked often, such as `restpp`, `gsql`, and `admin`, there are symbolic links that help you quickly get to the most recent log file of that category: +=== Third-Party Component Log Files -* `log.INFO` -** Contains regular output and errors -* `log.ERROR` -** Contains errors only -* `.out` -** Contains all output from the component process. Current `.out` logs have the form `.out`. -Historical logs have the form `-old-YYYY-MM-DDTHH-MM-SS.fff.out` +TigerGraph uses several open-source components (such as Kafka, Nginx, ZooKeeper, Kafkaconn, Kafkastream) that maintain their own log conventions. +* *NGINX Logs:* The NGINX log files (e.g., `nginx.out`, `nginx.error.log`, `nginx.access.log`) are generated directly by the NGINX web server itself and are not internal TigerGraph component logs. + +* *Kafka Logs:* Kafka logs include `controller.log`, `kafka.log`, `kafka-request.log`, `state-change.log`, and `server.log`. + +* *ZooKeeper Logs:* ZooKeeper logs are typically found as `zookeeper.out.*` in the ZooKeeper directory. + +== TigerGraph log structure + +[CAUTION] +==== +Log formats may differ between `.out` and `.INFO` logs and between different TigerGraph services. +==== + +* `log.INFO`: Contains regular output and errors. +* `log.ERROR`: Contains errors only. +* `.out`: Contains all output from the component process. Current `.out` logs have the form `.out`. Historical logs have the form `-old-YYYY-MM-DDTHH-MM-SS.fff.out` * `log.WARNING` or `log.DEBUG` -** `log.WARNING` contains warnings and all error level messages -* `log.FATAL` -** Contains outputs for any fatal level events +** `log.WARNING` contains warnings and all error-level messages. +** `log.DEBUG` contains debug-level messages (not created by default). +* `log.FATAL`: Contains outputs for any fatal level events [NOTE] +==== All services do not create a `log.DEBUG` file by default. To change this, modify the parameter `.BasicConfig.LogConfig.LogLevel`. -For example, `GSQL.BasicConfig.LogConfig.LogLevel`. See xref:reference:configuration-parameters.adoc[] for more information. +For example, `GSQL.BasicConfig.LogConfig.LogLevel`. See xref:reference:configuration-parameters.adoc[Configuration Parameters] for more information. +==== == Log locations on a cluster @@ -100,18 +160,8 @@ I@20210709 13:56:52.220 (SessionManager.java:204) All sessions aborted. I@20210709 13:56:52.224 (GsqlHAHandler.java:283) switched to new leader m1 ---- - -== Open source TigerGraph components - -The open source components that TigerGraph includes (Kafka, Nginx, ZooKeeper, Kafkaconn, Kafkastream) follow their respective logging behavior instead of having an `INFO/WARNING/ERROR` log, in addition to having an `.out` file for process output redirection. -For example, the Kafka logs have a `controller.log`, `kafka.log`, `kafka-request.log`, `state-change.log`, and `server.log`. - == Log rotation TigerGraph also handles log rotation. -When the log is rotated, the log.LEVEL symlink is updated to point to the newest log. -The default configuration is to rotate under any of the following circumstances: - -* Log file max size exceeds 100mb -* Log is older than 90 days -* There are more than 100 files for that service +When a log is rotated, the symlink (e.g., `log.INFO`) is updated to point to the newest log file. +Logs are rotated when the file size exceeds *100 MB*, the log is older than *90 days*, or more than *100 files* exist for that service.