springcloud优雅停止上下线与熔断

shouldUseReadOnlyResponseCache 读取的是 eureka.server.use-read-only-response-cache 参数

responseCacheUpdateIntervalMs 读取的是 eureka.server.response-cache-update-interval-ms 参数

responseCacheAutoExpirationInSeconds 读取的是 eureka.server.response-cache-update-interval-ms 参数

eureka.server.use-read-only-response-cache: false

eureka.server.eviction-interval-timer-in-ms: 3000

eureka.instance.lease-expiration-duration-in-seconds: 15

注意：Eureka Server 一定要设置 eureka.server.eviction-interval-timer-in-ms ，否则这个配置无效，这个配置一般为服务刷新时间配置的三倍。默认 90s!

eureka.instance.lease-renewal-interval-in-seconds: 5

eureka.client.registryFetchIntervalSeconds: 5

ribbon.ServerListRefreshInterval: 5000

这些超时时间相互影响，三个地方都需要配置，一不小心就会出现服务不下线，服务不上线的囧境。不得不说 SpringCloud 的这套默认参数很复杂，一定要小心谨慎地修改。

org.springframework.retryspring-retry

ribbon.OkToRetryOnAllOperations:true # (是否所有操作都重试，若false则仅get请求重试)

ribbon.MaxAutoRetriesNextServer:3 # (重试负载均衡其他实例最大重试次数,不含首次实例)

ribbon.MaxAutoRetries:1 # (同一实例最大重试次数,不含首次调用)

ribbon.ReadTimeout:30000

ribbon.ConnectTimeout:3000

ribbon.retryableStatusCodes:404,500,503 # (那些状态进行重试)

spring.cloud.loadbalancer.retry.enable:true # (重试开关)

package com.sudheera.playground.spring.testgracefullshutdown.config;

import java.util.concurrent.Executor;

import java.util.concurrent.ThreadPoolExecutor;

import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;

import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.springframework.boot.web.embedded.tomcat.TomcatConnectorCustomizer;

import org.springframework.context.ApplicationListener;

import org.springframework.context.event.ContextClosedEvent;

import org.springframework.stereotype.Component;

/**

 * Custom Tomcat protocol handler's executor service shutdown listner implementation to avoid spring application context

 * to shutdown before shutting down the tomcat server threads. This bean will do following when the application context

 * closed event occurred.

 *

 * 1. Pausing connector to stop accepting new connections

 * 2. Invoke shutdown of the ThreadPoolExecutor used in the Tomcat protocol handler

 * 3. Await until ThreadPoolExecutor is completely shutdown

 *

 * Read : https://github.com/spring-projects/spring-boot/issues/4657 for more information regarding the issue, this

 * implementation is based on `wilkinsona`s proposed solution in the issue thread

 */

@Component

public class TomcatGracefulShutdownListener implements TomcatConnectorCustomizer, ApplicationListener<ContextClosedEvent> {

    private static final Logger log = LoggerFactory.getLogger(TomcatGracefulShutdownListener.class);

    private static final int AWAIT_SECONDS = 100;

    private volatile Connector connector;

    @Override

    public void customize(Connector connector) {

        this.connector = connector;

    }

    @Override

    public void onApplicationEvent(ContextClosedEvent event) {

        connector.pause();

        Executor executor = connector.getProtocolHandler().getExecutor();

        if (executor == null) {

            return;

        }

        if (!(executor instanceof ThreadPoolExecutor)) {

            log.warn("{} is not an instance of ThreadPoolExecutor class, Can't ensure graceful shutdown..!!", executor.getClass());

            return;

        }

        ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;

        log.info("Shutting down Tomcat thread pool executor. Active thread count : {}", threadPoolExecutor.getActiveCount());

        threadPoolExecutor.shutdown();

        try {

            if (threadPoolExecutor.awaitTermination(AWAIT_SECONDS, TimeUnit.SECONDS)) {

                log.info("Tomcat shutdown gracefully.!");

                return;

            }

            threadPoolExecutor.shutdownNow();

            if (!threadPoolExecutor.awaitTermination(AWAIT_SECONDS, TimeUnit.SECONDS)) {

                log.warn("Tomcat thread pool executor did not terminate. Active thread count : {}", threadPoolExecutor.getActiveCount());

            }

        } catch (InterruptedException ex) {

            log.error("Got interrupted while waiting for graceful shutdown", ex);

            Thread.currentThread().interrupt();

        }

    }

}

# 开启优雅停止 Web 容器，默认为 IMMEDIATE：立即停止

server:

  shutdown: graceful

# 最大等待时间

spring:

  lifecycle:

    timeout-per-shutdown-phase: 30s

    org.springframework.boot

    spring-boot-starter-actuator

# 暴露 shutdown 接口

management:

  endpoint:

    shutdown:

      enabled: true

  endpoints:

    web:

      exposure:

        include: shutdown

@RestController

public class WorkController {

    @GetMapping("/work")

    public String work() throws InterruptedException {

        // 模拟复杂业务耗时处理流程

        Thread.sleep(10 * 1000L);

        return "success";

    }

}

{

    "message": "Shutting down, bye..."

}

2020-05-20 23:05:15.163  INFO 102724 --- [     Thread-253] o.s.b.w.e.tomcat.GracefulShutdown        : Commencing graceful shutdown. Waiting for active requests to complete

2020-05-20 23:05:15.287  INFO 102724 --- [tomcat-shutdown] o.s.b.w.e.tomcat.GracefulShutdown        : Graceful shutdown complete

2020-05-20 23:05:15.295  INFO 102724 --- [     Thread-253] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'applicationTaskExecutor'

@FunctionalInterface

public interface TomcatConnectorCustomizer {

    void customize(Connector connector);

}

@Bean

public GracefulShutdown gracefulShutdown() {

    return new GracefulShutdown();

}

private static class GracefulShutdown implements TomcatConnectorCustomizer, ApplicationListener {

    private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

    private volatile Connector connector;

    @Override

    public void customize(Connector connector) {

        this.connector = connector;

    }

    @Override

    public void onApplicationEvent(ContextClosedEvent event) {

        this.connector.pause();

        Executor executor = this.connector.getProtocolHandler().getExecutor();

        if (executor instanceof ThreadPoolExecutor) {

            try {

                ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;

                threadPoolExecutor.shutdown();

                if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {

                    log.warn("Tomcat thread pool did not shut down gracefully within 30 seconds. Proceeding with forceful shutdown");

                }

            } catch (InterruptedException ex) {

                Thread.currentThread().interrupt();

            }

        }

    }

}

@Bean

public ConfigurableServletWebServerFactory tomcatCustomizer() {

    TomcatServletWebServerFactory factory = new TomcatServletWebServerFactory();

    factory.addConnectorCustomizers(gracefulShutdown());

    return factory;

}

ribbon:

  ReadTimeout: 30000

  ConnectTimeout: 30000

feign:

  hystrix:

    # feign熔断器开关

    enabled: true

hystrix:

  command:

    default:

      execution:

        isolation:

          thread:

            #断路器的超时时间ms,默认1000

            timeoutInMilliseconds: 30000

      circuitBreaker:

        #当在配置时间窗口内达到此数量的失败后，进行短路

        requestVolumeThreshold: 20

        #出错百分比阈值，当达到此阈值后，开始短路。默认50%）

        errorThresholdPercentage: 50%

        #短路多久以后开始尝试是否恢复，默认5s）-单位ms

        sleepWindowInMilliseconds: 500

hystrix.command.default和hystrix.threadpool.default中的default为默认CommandKey

Command Properties

Execution相关的属性的配置：

hystrix.command.default.execution.isolation.strategy 隔离策略，默认是Thread, 可选Thread｜Semaphore

hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds 命令执行超时时间，默认1000ms

hystrix.command.default.execution.timeout.enabled 执行是否启用超时，默认启用true

hystrix.command.default.execution.isolation.thread.interruptOnTimeout 发生超时是是否中断，默认true

hystrix.command.default.execution.isolation.semaphore.maxConcurrentRequests 最大并发请求数，默认10，该参数当使用ExecutionIsolationStrategy.SEMAPHORE策略时才有效。如果达到最大并发请求数，请求会被拒绝。理论上选择semaphore size的原则和选择thread size一致，但选用semaphore时每次执行的单元要比较小且执行速度快（ms级别），否则的话应该用thread。

semaphore应该占整个容器（tomcat）的线程池的一小部分。

Fallback相关的属性

这些参数可以应用于Hystrix的THREAD和SEMAPHORE策略

hystrix.command.default.fallback.isolation.semaphore.maxConcurrentRequests 如果并发数达到该设置值，请求会被拒绝和抛出异常并且fallback不会被调用。默认10

hystrix.command.default.fallback.enabled 当执行失败或者请求被拒绝，是否会尝试调用hystrixCommand.getFallback() 。默认true

Circuit Breaker相关的属性

hystrix.command.default.circuitBreaker.enabled 用来跟踪circuit的健康性，如果未达标则让request短路。默认true

hystrix.command.default.circuitBreaker.requestVolumeThreshold 一个rolling window内最小的请求数。如果设为20，那么当一个rolling window的时间内（比如说1个rolling window是10秒）收到19个请求，即使19个请求都失败，也不会触发circuit break。默认20

hystrix.command.default.circuitBreaker.sleepWindowInMilliseconds 触发短路的时间值，当该值设为5000时，则当触发circuit break后的5000毫秒内都会拒绝request，也就是5000毫秒后才会关闭circuit。默认5000

hystrix.command.default.circuitBreaker.errorThresholdPercentage错误比率阀值，如果错误率>=该值，circuit会被打开，并短路所有请求触发fallback。默认50

hystrix.command.default.circuitBreaker.forceOpen 强制打开熔断器，如果打开这个开关，那么拒绝所有request，默认false

hystrix.command.default.circuitBreaker.forceClosed 强制关闭熔断器 如果这个开关打开，circuit将一直关闭且忽略circuitBreaker.errorThresholdPercentage

Metrics相关参数

hystrix.command.default.metrics.rollingStats.timeInMilliseconds 设置统计的时间窗口值的，毫秒值，circuit break 的打开会根据1个rolling window的统计来计算。若rolling window被设为10000毫秒，则rolling window会被分成n个buckets，每个bucket包含success，failure，timeout，rejection的次数的统计信息。默认10000

hystrix.command.default.metrics.rollingStats.numBuckets 设置一个rolling window被划分的数量，若numBuckets＝10，rolling window＝10000，那么一个bucket的时间即1秒。必须符合rolling window % numberBuckets == 0。默认10

hystrix.command.default.metrics.rollingPercentile.enabled 执行时是否enable指标的计算和跟踪，默认true

hystrix.command.default.metrics.rollingPercentile.timeInMilliseconds 设置rolling percentile window的时间，默认60000

hystrix.command.default.metrics.rollingPercentile.numBuckets 设置rolling percentile window的numberBuckets。逻辑同上。默认6

hystrix.command.default.metrics.rollingPercentile.bucketSize 如果bucket size＝100，window＝10s，若这10s里有500次执行，只有最后100次执行会被统计到bucket里去。增加该值会增加内存开销以及排序的开销。默认100

hystrix.command.default.metrics.healthSnapshot.intervalInMilliseconds 记录health 快照（用来统计成功和错误绿）的间隔，默认500ms

Request Context 相关参数

hystrix.command.default.requestCache.enabled 默认true，需要重载getCacheKey()，返回null时不缓存

hystrix.command.default.requestLog.enabled 记录日志到HystrixRequestLog，默认true

Collapser Properties 相关参数

hystrix.collapser.default.maxRequestsInBatch 单次批处理的最大请求数，达到该数量触发批处理，默认Integer.MAX_VALUE

hystrix.collapser.default.timerDelayInMilliseconds 触发批处理的延迟，也可以为创建批处理的时间＋该值，默认10

hystrix.collapser.default.requestCache.enabled 是否对HystrixCollapser.execute() and HystrixCollapser.queue()的cache，默认true

ThreadPool 相关参数

线程数默认值10适用于大部分情况（有时可以设置得更小），如果需要设置得更大，那有个基本得公式可以follow：

requests per second at peak when healthy × 99th percentile latency in seconds + some breathing room

每秒最大支撑的请求数 (99%平均响应时间 + 缓存值)

比如：每秒能处理1000个请求，99%的请求响应时间是60ms，那么公式是：

1000 （0.060+0.012）

基本得原则时保持线程池尽可能小，他主要是为了释放压力，防止资源被阻塞。

当一切都是正常的时候，线程池一般仅会有1到2个线程激活来提供服务

hystrix.threadpool.default.coreSize 并发执行的最大线程数，默认10

hystrix.threadpool.default.maxQueueSize BlockingQueue的最大队列数，当设为－1，会使用SynchronousQueue，值为正时使用LinkedBlcokingQueue。该设置只会在初始化时有效，之后不能修改threadpool的queue size，除非reinitialising thread executor。默认－1。

hystrix.threadpool.default.queueSizeRejectionThreshold 即使maxQueueSize没有达到，达到queueSizeRejectionThreshold该值后，请求也会被拒绝。因为maxQueueSize不能被动态修改，这个参数将允许我们动态设置该值。if maxQueueSize == -1，该字段将不起作用

hystrix.threadpool.default.keepAliveTimeMinutes 如果corePoolSize和maxPoolSize设成一样（默认实现）该设置无效。如果通过plugin（https://github.com/Netflix/Hystrix/wiki/Plugins）使用自定义实现，该设置才有用，默认1.

hystrix.threadpool.default.metrics.rollingStats.timeInMilliseconds 线程池统计指标的时间，默认10000

hystrix.threadpool.default.metrics.rollingStats.numBuckets 将rolling window划分为n个buckets，默认10