Logging with ElasticSearch, Kibana, ASP.NET Core and Docker

提示：扫一扫查出行
【扫一扫了解最新限行尾号】
复制提示

好久不见，前两周经历了人生第一次"伪牛市"，基金和股市大起大落，更加坚信“你永远赚不到超出你认知范围之外的钱,除非靠着运气”，老韭菜诚不欺我也。

当能力与野心不匹配，只能多看书，收割那些不求甚解的韭菜。

言归正传，本文聊一聊在ASP.NET Core3.1中集成ElasticSearch、Kibana日志查询系统。

我们为什么需要ElasticSearch、Kibana？

成熟的应用程序，会在各个阶段以各种姿势记录日志，这些日志包括(但不限于)nginx日志、pod/Container、业务日志。

ElasticSearch是一个高度可扩展的全文搜索和分析引擎，使任何类型的日志记录变得容易，可访问和可搜索。ElasticSearch令人难以置信的速度和简单的查询语言，再加上Kibana的界面和图形，构成了功能强大的日志存储和查询搭档。

1.在Docker中启动ElasticSearch、Kibana

docker中启动Elastcisearch、kibana最简单的方式是创建docker-compose文件，将原服务侦听端口映射到宿主机端口。

version: '3.1'

services:

  elasticsearch:

   container_name: elasticsearch

   image: elasticsearch:7.6.2

   ports:

    - 9200:9200

   volumes:

    - elasticsearch-data:/usr/share/elasticsearch/data

   environment:

    - xpack.monitoring.enabled=true

    - xpack.watcher.enabled=false

    - "ES_JAVA_OPTS=-Xms1g -Xmx1g"

    - discovery.type=single-node

   networks:

    - elastic

  kibana:

   container_name: kibana

   image: kibana:7.6.2

   ports:

    - 5601:5601

   depends_on:

    - elasticsearch

   environment:

    - ELASTICSEARCH_URL=http://localhost:9200

   networks:

    - elastic

networks:

  elastic:

    driver: bridge

volumes:

  elasticsearch-data:

在后台拉取镜像，启动容器：

docker-compose  up -d

确认ElasticSearch,kibana启动成功：

2. 使用ASP.NET Core和NLog向ES发送日志

VS Code创建.NetCore程序

dotnet new mvc --no-https -o Elastic.Kibana.NLog

cd Elastic.Kibana.NLog

添加NLog依赖包

dotnet add package NLog

dotnet add package NLog.Web.AspNetCore

dotnet add package NLog.Targets.ElasticSearch

添加nlog.config文件

<?xml version="1.0" encoding="utf-8" ?>

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      autoReload="true"

      internalLogLevel="Info"

      internalLogFile="internal-nlog.txt">

  <!-- enable asp.net core layout renderers -->

  <extensions>

    <add assembly="NLog.Web.AspNetCore"/>

  </extensions>

  <!-- the targets to write to -->

  <targets>

    <!-- write logs to file  -->

    <target xsi:type="File" name="allfile" fileName="/home/root/nlog-all-${shortdate}.log"

            layout="${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}" />

    <!-- another file log, only own logs. Uses some ASP.NET core renderers -->

    <target xsi:type="File" name="ownFile-web" fileName="/home/root/nlog-own-${shortdate}.log"

            layout="${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}" />

    <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">

      <target xsi:type="ElasticSearch" uri="http://localhost:9200/" />

    </target>

  </targets>

  <!-- rules to map from logger name to target -->

  <rules>

    <!--All logs, including from Microsoft-->

    <logger name="*" minlevel="Trace" writeTo="allfile" />

    <!--Skip non-critical Microsoft logs and so log only own logs-->

    <logger name="Microsoft.*" maxlevel="Info" final="true" />

    <!-- BlackHole without writeTo -->

    <logger name="*" minlevel="Trace" writeTo="ownFile-web" />

    <logger name="*" minlevel="Info" writeTo="elastic" />

  </rules>

</nlog>

ASP.NET Core添加Nlog并输出日志

    public class Program

    {

        public static void Main(string[] args)

        {

            var webHost = WebHost.CreateDefaultBuilder(args)

                     .ConfigureLogging((hostingContext, loggingBuilder) =>

                     {

                         loggingBuilder.AddConsole(x => x.IncludeScopes = true).AddDebug();

                     })

                     .UseNLog()

                     .UseStartup<Startup>()

                     .Build();

            webHost.Run();

        }

    }

Controller自定义日志：

public class HomeController : Controller

    {

        private readonly ILogger<HomeController> _logger;

        public HomeController(ILogger<HomeController> logger)

        {

            _logger = logger;

        }

        public IActionResult Index()

        {

           _logger.LogInformation("HomeController Index executed at {date}", DateTime.UtcNow);

            return View();

        }

        public IActionResult Privacy()

        {

            return View();

        }

        [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]

        public IActionResult Error()

        {

            return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });

        }

    }

3.在kibana中创建Index Pattern

dotnet run启动ASP.NET Core程序，现在开始产生日志。

Kibana使用 index pattern 从 ElasticSearch indices

中获取数据。要在Kibana中显示日志，需要先定义index pattern:

之后选择 @timestamp 时间过滤。